2023-04-03 13:56:10,211 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27 2023-04-03 13:56:10,225 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-03 13:56:10,240 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-03 13:56:10,240 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/cluster_f53e868d-306e-b331-2b8a-fd0a76147c40, deleteOnExit=true 2023-04-03 13:56:10,240 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-03 13:56:10,241 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/test.cache.data in system properties and HBase conf 2023-04-03 13:56:10,241 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/hadoop.tmp.dir in system properties and HBase conf 2023-04-03 13:56:10,242 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/hadoop.log.dir in system properties and HBase conf 2023-04-03 13:56:10,242 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-03 13:56:10,243 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-03 13:56:10,243 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-03 13:56:10,380 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-03 13:56:10,802 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-03 13:56:10,808 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-03 13:56:10,808 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-03 13:56:10,809 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-03 13:56:10,809 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-03 13:56:10,810 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-03 13:56:10,810 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-03 13:56:10,811 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-03 13:56:10,811 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-03 13:56:10,811 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-03 13:56:10,812 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/nfs.dump.dir in system properties and HBase conf 2023-04-03 13:56:10,812 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/java.io.tmpdir in system properties and HBase conf 2023-04-03 13:56:10,812 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-03 13:56:10,812 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-03 13:56:10,813 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-03 13:56:11,328 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-03 13:56:11,331 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-03 13:56:11,559 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-03 13:56:11,689 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-03 13:56:11,701 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-03 13:56:11,732 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-03 13:56:11,757 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/java.io.tmpdir/Jetty_localhost_localdomain_45913_hdfs____.jfq6ng/webapp 2023-04-03 13:56:11,856 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:45913 2023-04-03 13:56:11,884 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-03 13:56:11,885 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-03 13:56:12,207 WARN [Listener at localhost.localdomain/35003] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-03 13:56:12,288 WARN [Listener at localhost.localdomain/35003] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-03 13:56:12,303 WARN [Listener at localhost.localdomain/35003] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-03 13:56:12,308 INFO [Listener at localhost.localdomain/35003] log.Slf4jLog(67): jetty-6.1.26 2023-04-03 13:56:12,312 INFO [Listener at localhost.localdomain/35003] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/java.io.tmpdir/Jetty_localhost_33263_datanode____llm9va/webapp 2023-04-03 13:56:12,385 INFO [Listener at localhost.localdomain/35003] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33263 2023-04-03 13:56:12,644 WARN [Listener at localhost.localdomain/43477] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-03 13:56:12,955 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7c9b42c7acf65548: Processing first storage report for DS-72f33569-33f3-44e3-ad7a-55f3f4193ab6 from datanode 5db87ce9-8a9b-49d5-8f71-6cbe87f3a0d0 2023-04-03 13:56:12,957 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7c9b42c7acf65548: from storage DS-72f33569-33f3-44e3-ad7a-55f3f4193ab6 node DatanodeRegistration(127.0.0.1:43521, datanodeUuid=5db87ce9-8a9b-49d5-8f71-6cbe87f3a0d0, infoPort=34717, infoSecurePort=0, ipcPort=43477, storageInfo=lv=-57;cid=testClusterID;nsid=2145057547;c=1680530171395), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-03 13:56:12,957 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7c9b42c7acf65548: Processing first storage report for DS-ce9427e9-67ff-45ef-b5b4-4d6303e12e6e from datanode 5db87ce9-8a9b-49d5-8f71-6cbe87f3a0d0 2023-04-03 13:56:12,957 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7c9b42c7acf65548: from storage DS-ce9427e9-67ff-45ef-b5b4-4d6303e12e6e node DatanodeRegistration(127.0.0.1:43521, datanodeUuid=5db87ce9-8a9b-49d5-8f71-6cbe87f3a0d0, infoPort=34717, infoSecurePort=0, ipcPort=43477, storageInfo=lv=-57;cid=testClusterID;nsid=2145057547;c=1680530171395), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-03 13:56:13,146 DEBUG [Listener at localhost.localdomain/43477] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27 2023-04-03 13:56:13,204 INFO [Listener at localhost.localdomain/43477] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/cluster_f53e868d-306e-b331-2b8a-fd0a76147c40/zookeeper_0, clientPort=57081, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/cluster_f53e868d-306e-b331-2b8a-fd0a76147c40/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/cluster_f53e868d-306e-b331-2b8a-fd0a76147c40/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-03 13:56:13,216 INFO [Listener at localhost.localdomain/43477] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57081 2023-04-03 13:56:13,225 INFO [Listener at localhost.localdomain/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 13:56:13,227 INFO [Listener at localhost.localdomain/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 13:56:13,836 INFO [Listener at localhost.localdomain/43477] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156 with version=8 2023-04-03 13:56:13,837 INFO [Listener at localhost.localdomain/43477] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/hbase-staging 2023-04-03 13:56:14,078 INFO [Listener at localhost.localdomain/43477] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-03 13:56:14,425 INFO [Listener at localhost.localdomain/43477] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-04-03 13:56:14,450 INFO [Listener at localhost.localdomain/43477] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 13:56:14,450 INFO [Listener at localhost.localdomain/43477] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-03 13:56:14,450 INFO [Listener at localhost.localdomain/43477] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-03 13:56:14,450 INFO [Listener at localhost.localdomain/43477] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 13:56:14,451 INFO [Listener at localhost.localdomain/43477] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-03 13:56:14,561 INFO [Listener at localhost.localdomain/43477] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-03 13:56:14,619 DEBUG [Listener at localhost.localdomain/43477] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-03 13:56:14,692 INFO [Listener at localhost.localdomain/43477] ipc.NettyRpcServer(120): Bind to /148.251.75.209:37337 2023-04-03 13:56:14,702 INFO [Listener at localhost.localdomain/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 13:56:14,704 INFO [Listener at localhost.localdomain/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 13:56:14,720 INFO [Listener at localhost.localdomain/43477] zookeeper.RecoverableZooKeeper(93): Process identifier=master:37337 connecting to ZooKeeper ensemble=127.0.0.1:57081 2023-04-03 13:56:14,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:373370x0, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-03 13:56:14,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:37337-0x100771d85340000 connected 2023-04-03 13:56:14,777 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 13:56:14,778 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 13:56:14,781 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-03 13:56:14,791 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37337 2023-04-03 13:56:14,792 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37337 2023-04-03 13:56:14,792 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37337 2023-04-03 13:56:14,792 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37337 2023-04-03 13:56:14,793 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37337 2023-04-03 13:56:14,800 INFO [Listener at localhost.localdomain/43477] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156, hbase.cluster.distributed=false 2023-04-03 13:56:14,857 INFO [Listener at localhost.localdomain/43477] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-04-03 13:56:14,857 INFO [Listener at localhost.localdomain/43477] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 13:56:14,857 INFO [Listener at localhost.localdomain/43477] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-03 13:56:14,857 INFO [Listener at localhost.localdomain/43477] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-03 13:56:14,857 INFO [Listener at localhost.localdomain/43477] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 13:56:14,857 INFO [Listener at localhost.localdomain/43477] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-03 13:56:14,861 INFO [Listener at localhost.localdomain/43477] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-03 13:56:14,864 INFO [Listener at localhost.localdomain/43477] ipc.NettyRpcServer(120): Bind to /148.251.75.209:39695 2023-04-03 13:56:14,866 INFO [Listener at localhost.localdomain/43477] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-03 13:56:14,871 DEBUG [Listener at localhost.localdomain/43477] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-03 13:56:14,872 INFO [Listener at localhost.localdomain/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 13:56:14,875 INFO [Listener at localhost.localdomain/43477] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 13:56:14,876 INFO [Listener at localhost.localdomain/43477] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39695 connecting to ZooKeeper ensemble=127.0.0.1:57081 2023-04-03 13:56:14,880 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:396950x0, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-03 13:56:14,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39695-0x100771d85340001 connected 2023-04-03 13:56:14,881 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 13:56:14,882 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 13:56:14,883 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-03 13:56:14,883 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39695 2023-04-03 13:56:14,883 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39695 2023-04-03 13:56:14,884 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39695 2023-04-03 13:56:14,884 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39695 2023-04-03 13:56:14,884 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39695 2023-04-03 13:56:14,886 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:56:14,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-03 13:56:14,895 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:56:14,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-03 13:56:14,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-03 13:56:14,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:56:14,912 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-03 13:56:14,913 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,37337,1680530173952 from backup master directory 2023-04-03 13:56:14,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-03 13:56:14,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:56:14,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-03 13:56:14,916 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-03 13:56:14,917 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:56:14,919 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-03 13:56:14,921 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-03 13:56:14,996 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/hbase.id with ID: 58828d35-9132-4a61-be50-326ebb8126ad 2023-04-03 13:56:15,037 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 13:56:15,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:56:15,090 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x07ff1af1 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:15,119 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@579c2551, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:15,138 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:56:15,139 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-03 13:56:15,154 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-03 13:56:15,154 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-03 13:56:15,156 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:15,159 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:15,160 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 13:56:15,188 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store-tmp 2023-04-03 13:56:15,618 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:15,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-03 13:56:15,619 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 13:56:15,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 13:56:15,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-03 13:56:15,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 13:56:15,619 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 13:56:15,619 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 13:56:15,621 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/WALs/jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:56:15,639 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C37337%2C1680530173952, suffix=, logDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/WALs/jenkins-hbase20.apache.org,37337,1680530173952, archiveDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/oldWALs, maxLogs=10 2023-04-03 13:56:15,694 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43521,DS-72f33569-33f3-44e3-ad7a-55f3f4193ab6,DISK] 2023-04-03 13:56:15,700 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:15,751 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/WALs/jenkins-hbase20.apache.org,37337,1680530173952/jenkins-hbase20.apache.org%2C37337%2C1680530173952.1680530175647 2023-04-03 13:56:15,752 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43521,DS-72f33569-33f3-44e3-ad7a-55f3f4193ab6,DISK]] 2023-04-03 13:56:15,752 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:56:15,753 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:15,755 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 13:56:15,756 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 13:56:15,807 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-03 13:56:15,813 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-03 13:56:15,832 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-03 13:56:15,842 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:15,848 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-03 13:56:15,850 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-03 13:56:15,865 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 13:56:15,869 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:56:15,870 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64252721, jitterRate=-0.042559847235679626}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-03 13:56:15,870 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 13:56:15,871 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-03 13:56:15,889 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-03 13:56:15,889 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-03 13:56:15,891 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-03 13:56:15,892 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-03 13:56:15,918 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 25 msec 2023-04-03 13:56:15,918 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-03 13:56:15,940 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-03 13:56:15,944 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-03 13:56:15,964 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-03 13:56:15,967 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-03 13:56:15,969 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-03 13:56:15,973 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-03 13:56:15,977 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-03 13:56:15,979 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:56:15,981 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-03 13:56:15,981 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-03 13:56:15,992 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-03 13:56:15,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-03 13:56:15,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-03 13:56:15,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:56:15,996 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,37337,1680530173952, sessionid=0x100771d85340000, setting cluster-up flag (Was=false) 2023-04-03 13:56:16,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:56:16,014 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-03 13:56:16,015 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:56:16,019 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:56:16,022 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-03 13:56:16,023 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:56:16,026 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.hbase-snapshot/.tmp 2023-04-03 13:56:16,089 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(952): ClusterId : 58828d35-9132-4a61-be50-326ebb8126ad 2023-04-03 13:56:16,092 DEBUG [RS:0;jenkins-hbase20:39695] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-03 13:56:16,097 DEBUG [RS:0;jenkins-hbase20:39695] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-03 13:56:16,097 DEBUG [RS:0;jenkins-hbase20:39695] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-03 13:56:16,099 DEBUG [RS:0;jenkins-hbase20:39695] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-03 13:56:16,100 DEBUG [RS:0;jenkins-hbase20:39695] zookeeper.ReadOnlyZKClient(139): Connect 0x4d1034ff to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:16,109 DEBUG [RS:0;jenkins-hbase20:39695] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43d7c32d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:16,110 DEBUG [RS:0;jenkins-hbase20:39695] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e85f8f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-03 13:56:16,127 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-03 13:56:16,131 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:39695 2023-04-03 13:56:16,134 INFO [RS:0;jenkins-hbase20:39695] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-03 13:56:16,135 INFO [RS:0;jenkins-hbase20:39695] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-03 13:56:16,135 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1023): About to register with Master. 2023-04-03 13:56:16,135 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-03 13:56:16,136 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-03 13:56:16,136 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-03 13:56:16,136 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-04-03 13:56:16,136 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-04-03 13:56:16,136 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,136 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-03 13:56:16,136 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,137 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,37337,1680530173952 with isa=jenkins-hbase20.apache.org/148.251.75.209:39695, startcode=1680530174856 2023-04-03 13:56:16,139 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680530206139 2023-04-03 13:56:16,141 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-03 13:56:16,142 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-03 13:56:16,142 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-03 13:56:16,147 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-03 13:56:16,152 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-03 13:56:16,155 DEBUG [RS:0;jenkins-hbase20:39695] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-03 13:56:16,158 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-03 13:56:16,158 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-03 13:56:16,159 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-03 13:56:16,159 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-03 13:56:16,159 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,161 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-03 13:56:16,162 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-03 13:56:16,163 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-03 13:56:16,165 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-03 13:56:16,165 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-03 13:56:16,167 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680530176167,5,FailOnTimeoutGroup] 2023-04-03 13:56:16,167 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680530176167,5,FailOnTimeoutGroup] 2023-04-03 13:56:16,167 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,167 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-03 13:56:16,169 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,170 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,179 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-03 13:56:16,180 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-03 13:56:16,181 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156 2023-04-03 13:56:16,211 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52055, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-03 13:56:16,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:16,233 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156 2023-04-03 13:56:16,233 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:35003 2023-04-03 13:56:16,233 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-03 13:56:16,237 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 13:56:16,239 DEBUG [RS:0;jenkins-hbase20:39695] zookeeper.ZKUtil(162): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:16,239 WARN [RS:0;jenkins-hbase20:39695] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-03 13:56:16,240 INFO [RS:0;jenkins-hbase20:39695] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 13:56:16,240 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,39695,1680530174856] 2023-04-03 13:56:16,240 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/WALs/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:16,250 DEBUG [RS:0;jenkins-hbase20:39695] zookeeper.ZKUtil(162): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:16,261 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-03 13:56:16,267 INFO [RS:0;jenkins-hbase20:39695] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-03 13:56:16,283 INFO [RS:0;jenkins-hbase20:39695] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-03 13:56:16,286 INFO [RS:0;jenkins-hbase20:39695] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-03 13:56:16,286 INFO [RS:0;jenkins-hbase20:39695] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,287 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-03 13:56:16,293 INFO [RS:0;jenkins-hbase20:39695] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,293 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,294 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,294 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,294 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,294 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,294 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-04-03 13:56:16,294 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,295 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,295 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,295 DEBUG [RS:0;jenkins-hbase20:39695] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-04-03 13:56:16,296 INFO [RS:0;jenkins-hbase20:39695] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,296 INFO [RS:0;jenkins-hbase20:39695] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,296 INFO [RS:0;jenkins-hbase20:39695] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,310 INFO [RS:0;jenkins-hbase20:39695] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-03 13:56:16,313 INFO [RS:0;jenkins-hbase20:39695] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,39695,1680530174856-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:16,326 INFO [RS:0;jenkins-hbase20:39695] regionserver.Replication(203): jenkins-hbase20.apache.org,39695,1680530174856 started 2023-04-03 13:56:16,326 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,39695,1680530174856, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:39695, sessionid=0x100771d85340001 2023-04-03 13:56:16,327 DEBUG [RS:0;jenkins-hbase20:39695] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-03 13:56:16,327 DEBUG [RS:0;jenkins-hbase20:39695] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:16,327 DEBUG [RS:0;jenkins-hbase20:39695] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39695,1680530174856' 2023-04-03 13:56:16,327 DEBUG [RS:0;jenkins-hbase20:39695] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:16,328 DEBUG [RS:0;jenkins-hbase20:39695] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:16,328 DEBUG [RS:0;jenkins-hbase20:39695] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-03 13:56:16,328 DEBUG [RS:0;jenkins-hbase20:39695] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-03 13:56:16,328 DEBUG [RS:0;jenkins-hbase20:39695] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:16,328 DEBUG [RS:0;jenkins-hbase20:39695] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,39695,1680530174856' 2023-04-03 13:56:16,328 DEBUG [RS:0;jenkins-hbase20:39695] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-03 13:56:16,329 DEBUG [RS:0;jenkins-hbase20:39695] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-03 13:56:16,329 DEBUG [RS:0;jenkins-hbase20:39695] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-03 13:56:16,329 INFO [RS:0;jenkins-hbase20:39695] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-03 13:56:16,330 INFO [RS:0;jenkins-hbase20:39695] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-03 13:56:16,447 INFO [RS:0;jenkins-hbase20:39695] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39695%2C1680530174856, suffix=, logDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/WALs/jenkins-hbase20.apache.org,39695,1680530174856, archiveDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/oldWALs, maxLogs=32 2023-04-03 13:56:16,463 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43521,DS-72f33569-33f3-44e3-ad7a-55f3f4193ab6,DISK] 2023-04-03 13:56:16,467 INFO [RS:0;jenkins-hbase20:39695] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/WALs/jenkins-hbase20.apache.org,39695,1680530174856/jenkins-hbase20.apache.org%2C39695%2C1680530174856.1680530176449 2023-04-03 13:56:16,467 DEBUG [RS:0;jenkins-hbase20:39695] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43521,DS-72f33569-33f3-44e3-ad7a-55f3f4193ab6,DISK]] 2023-04-03 13:56:16,595 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:16,598 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-03 13:56:16,601 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/info 2023-04-03 13:56:16,602 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-03 13:56:16,603 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:16,603 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-03 13:56:16,606 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/rep_barrier 2023-04-03 13:56:16,606 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-03 13:56:16,607 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:16,608 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-03 13:56:16,610 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/table 2023-04-03 13:56:16,610 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-03 13:56:16,611 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:16,613 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740 2023-04-03 13:56:16,614 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740 2023-04-03 13:56:16,618 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:56:16,620 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-03 13:56:16,624 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:56:16,625 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74446796, jitterRate=0.10934370756149292}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:56:16,625 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-03 13:56:16,625 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-03 13:56:16,625 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-03 13:56:16,625 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-03 13:56:16,626 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-03 13:56:16,626 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-03 13:56:16,627 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-03 13:56:16,627 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-03 13:56:16,632 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-03 13:56:16,632 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-03 13:56:16,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-03 13:56:16,651 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-03 13:56:16,653 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-03 13:56:16,805 DEBUG [jenkins-hbase20:37337] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-03 13:56:16,808 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,39695,1680530174856, state=OPENING 2023-04-03 13:56:16,814 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-03 13:56:16,815 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:56:16,816 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-03 13:56:16,819 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:56:17,006 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:17,008 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-03 13:56:17,012 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36190, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-03 13:56:17,024 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-03 13:56:17,024 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 13:56:17,028 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C39695%2C1680530174856.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/WALs/jenkins-hbase20.apache.org,39695,1680530174856, archiveDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/oldWALs, maxLogs=32 2023-04-03 13:56:17,045 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43521,DS-72f33569-33f3-44e3-ad7a-55f3f4193ab6,DISK] 2023-04-03 13:56:17,049 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/WALs/jenkins-hbase20.apache.org,39695,1680530174856/jenkins-hbase20.apache.org%2C39695%2C1680530174856.meta.1680530177029.meta 2023-04-03 13:56:17,050 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43521,DS-72f33569-33f3-44e3-ad7a-55f3f4193ab6,DISK]] 2023-04-03 13:56:17,050 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:56:17,052 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-03 13:56:17,067 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-03 13:56:17,071 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-03 13:56:17,076 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-03 13:56:17,076 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:17,076 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-03 13:56:17,076 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-03 13:56:17,079 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-03 13:56:17,081 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/info 2023-04-03 13:56:17,081 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/info 2023-04-03 13:56:17,082 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-03 13:56:17,083 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:17,083 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-03 13:56:17,084 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/rep_barrier 2023-04-03 13:56:17,085 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/rep_barrier 2023-04-03 13:56:17,085 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-03 13:56:17,086 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:17,087 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-03 13:56:17,088 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/table 2023-04-03 13:56:17,088 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/table 2023-04-03 13:56:17,089 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-03 13:56:17,090 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:17,091 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740 2023-04-03 13:56:17,094 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740 2023-04-03 13:56:17,097 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:56:17,101 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-03 13:56:17,103 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71592844, jitterRate=0.06681650876998901}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:56:17,103 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-03 13:56:17,115 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680530176999 2023-04-03 13:56:17,135 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-03 13:56:17,136 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-03 13:56:17,137 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,39695,1680530174856, state=OPEN 2023-04-03 13:56:17,139 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-03 13:56:17,139 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-03 13:56:17,143 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-03 13:56:17,144 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,39695,1680530174856 in 321 msec 2023-04-03 13:56:17,148 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-03 13:56:17,149 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 505 msec 2023-04-03 13:56:17,154 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.0880 sec 2023-04-03 13:56:17,154 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680530177154, completionTime=-1 2023-04-03 13:56:17,154 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-03 13:56:17,154 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-03 13:56:17,209 DEBUG [hconnection-0x3e0d66be-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:17,214 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:17,230 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-03 13:56:17,231 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680530237231 2023-04-03 13:56:17,231 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680530297231 2023-04-03 13:56:17,231 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 76 msec 2023-04-03 13:56:17,252 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37337,1680530173952-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:17,252 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37337,1680530173952-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:17,252 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37337,1680530173952-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:17,254 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:37337, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:17,254 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-03 13:56:17,260 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-03 13:56:17,267 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-03 13:56:17,268 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-03 13:56:17,277 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-03 13:56:17,280 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 13:56:17,283 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 13:56:17,304 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:17,307 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd empty. 2023-04-03 13:56:17,308 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:17,308 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-03 13:56:17,733 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-03 13:56:17,736 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1162c80dd821d6d790c48c84de83a7bd, NAME => 'hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp 2023-04-03 13:56:17,751 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:17,752 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 1162c80dd821d6d790c48c84de83a7bd, disabling compactions & flushes 2023-04-03 13:56:17,752 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:56:17,752 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:56:17,752 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. after waiting 0 ms 2023-04-03 13:56:17,752 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:56:17,752 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:56:17,752 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 1162c80dd821d6d790c48c84de83a7bd: 2023-04-03 13:56:17,757 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 13:56:17,769 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680530177760"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530177760"}]},"ts":"1680530177760"} 2023-04-03 13:56:17,792 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 13:56:17,794 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 13:56:17,799 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530177795"}]},"ts":"1680530177795"} 2023-04-03 13:56:17,803 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-03 13:56:17,809 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1162c80dd821d6d790c48c84de83a7bd, ASSIGN}] 2023-04-03 13:56:17,812 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1162c80dd821d6d790c48c84de83a7bd, ASSIGN 2023-04-03 13:56:17,814 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=1162c80dd821d6d790c48c84de83a7bd, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=false 2023-04-03 13:56:17,965 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1162c80dd821d6d790c48c84de83a7bd, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:17,965 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680530177965"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530177965"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530177965"}]},"ts":"1680530177965"} 2023-04-03 13:56:17,970 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 1162c80dd821d6d790c48c84de83a7bd, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:56:18,130 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:56:18,131 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1162c80dd821d6d790c48c84de83a7bd, NAME => 'hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:56:18,132 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:18,132 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:18,132 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:18,133 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:18,135 INFO [StoreOpener-1162c80dd821d6d790c48c84de83a7bd-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:18,137 DEBUG [StoreOpener-1162c80dd821d6d790c48c84de83a7bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd/info 2023-04-03 13:56:18,137 DEBUG [StoreOpener-1162c80dd821d6d790c48c84de83a7bd-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd/info 2023-04-03 13:56:18,138 INFO [StoreOpener-1162c80dd821d6d790c48c84de83a7bd-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1162c80dd821d6d790c48c84de83a7bd columnFamilyName info 2023-04-03 13:56:18,139 INFO [StoreOpener-1162c80dd821d6d790c48c84de83a7bd-1] regionserver.HStore(310): Store=1162c80dd821d6d790c48c84de83a7bd/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:18,140 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:18,141 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:18,146 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:56:18,149 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:56:18,150 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1162c80dd821d6d790c48c84de83a7bd; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64505764, jitterRate=-0.038789212703704834}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-03 13:56:18,150 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1162c80dd821d6d790c48c84de83a7bd: 2023-04-03 13:56:18,152 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd., pid=6, masterSystemTime=1680530178124 2023-04-03 13:56:18,157 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:56:18,157 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:56:18,159 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1162c80dd821d6d790c48c84de83a7bd, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:18,160 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680530178158"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530178158"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530178158"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530178158"}]},"ts":"1680530178158"} 2023-04-03 13:56:18,166 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-03 13:56:18,166 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 1162c80dd821d6d790c48c84de83a7bd, server=jenkins-hbase20.apache.org,39695,1680530174856 in 193 msec 2023-04-03 13:56:18,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-03 13:56:18,170 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=1162c80dd821d6d790c48c84de83a7bd, ASSIGN in 357 msec 2023-04-03 13:56:18,171 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 13:56:18,172 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530178172"}]},"ts":"1680530178172"} 2023-04-03 13:56:18,175 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-03 13:56:18,178 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 13:56:18,180 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-03 13:56:18,181 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 909 msec 2023-04-03 13:56:18,181 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-03 13:56:18,181 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:56:18,219 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-03 13:56:18,237 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-03 13:56:18,242 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-03 13:56:18,254 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-03 13:56:18,267 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-03 13:56:18,272 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-04-03 13:56:18,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-03 13:56:18,287 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-03 13:56:18,287 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.370sec 2023-04-03 13:56:18,289 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-03 13:56:18,291 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-03 13:56:18,291 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-03 13:56:18,292 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37337,1680530173952-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-03 13:56:18,293 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,37337,1680530173952-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-03 13:56:18,294 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x7d5f4139 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:18,302 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e23d53f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:18,307 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-03 13:56:18,314 DEBUG [hconnection-0x5ae1b7fe-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:18,328 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36206, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:18,339 INFO [Listener at localhost.localdomain/43477] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:56:18,363 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=172, ProcessCount=166, AvailableMemoryMB=3493 2023-04-03 13:56:18,380 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 13:56:18,384 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 13:56:18,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:56:18,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:56:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:18,402 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 13:56:18,405 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 13:56:18,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-03 13:56:18,408 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:18,410 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908 empty. 2023-04-03 13:56:18,411 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:18,412 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:56:18,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 13:56:18,431 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:56:18,433 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3d9e5bf7331338c5e5778539b0a0c908, NAME => 'TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp 2023-04-03 13:56:18,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 13:56:18,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 13:56:18,846 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:18,847 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3d9e5bf7331338c5e5778539b0a0c908, disabling compactions & flushes 2023-04-03 13:56:18,847 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:18,847 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:18,847 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. after waiting 0 ms 2023-04-03 13:56:18,847 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:18,848 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:18,848 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:18,858 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 13:56:18,865 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530178864"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530178864"}]},"ts":"1680530178864"} 2023-04-03 13:56:18,871 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 13:56:18,874 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 13:56:18,874 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530178874"}]},"ts":"1680530178874"} 2023-04-03 13:56:18,878 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 13:56:18,882 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d9e5bf7331338c5e5778539b0a0c908, ASSIGN}] 2023-04-03 13:56:18,884 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d9e5bf7331338c5e5778539b0a0c908, ASSIGN 2023-04-03 13:56:18,886 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d9e5bf7331338c5e5778539b0a0c908, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=false 2023-04-03 13:56:19,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 13:56:19,038 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3d9e5bf7331338c5e5778539b0a0c908, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,039 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530179038"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530179038"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530179038"}]},"ts":"1680530179038"} 2023-04-03 13:56:19,042 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:56:19,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:19,209 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3d9e5bf7331338c5e5778539b0a0c908, NAME => 'TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:56:19,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:19,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,213 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,216 DEBUG [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A 2023-04-03 13:56:19,216 DEBUG [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A 2023-04-03 13:56:19,218 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:19,218 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d9e5bf7331338c5e5778539b0a0c908 columnFamilyName A 2023-04-03 13:56:19,219 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.HStore(310): Store=3d9e5bf7331338c5e5778539b0a0c908/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:19,220 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,222 DEBUG [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B 2023-04-03 13:56:19,222 DEBUG [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B 2023-04-03 13:56:19,223 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:19,223 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d9e5bf7331338c5e5778539b0a0c908 columnFamilyName B 2023-04-03 13:56:19,224 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.HStore(310): Store=3d9e5bf7331338c5e5778539b0a0c908/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:19,225 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,227 DEBUG [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C 2023-04-03 13:56:19,227 DEBUG [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C 2023-04-03 13:56:19,228 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:19,228 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3d9e5bf7331338c5e5778539b0a0c908 columnFamilyName C 2023-04-03 13:56:19,229 INFO [StoreOpener-3d9e5bf7331338c5e5778539b0a0c908-1] regionserver.HStore(310): Store=3d9e5bf7331338c5e5778539b0a0c908/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:19,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:19,233 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:56:19,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:56:19,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 3d9e5bf7331338c5e5778539b0a0c908; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74479535, jitterRate=0.1098315566778183}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:56:19,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:19,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., pid=11, masterSystemTime=1680530179196 2023-04-03 13:56:19,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:19,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:19,254 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=3d9e5bf7331338c5e5778539b0a0c908, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,254 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530179253"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530179253"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530179253"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530179253"}]},"ts":"1680530179253"} 2023-04-03 13:56:19,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-03 13:56:19,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 in 215 msec 2023-04-03 13:56:19,266 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-03 13:56:19,267 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d9e5bf7331338c5e5778539b0a0c908, ASSIGN in 380 msec 2023-04-03 13:56:19,268 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 13:56:19,268 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530179268"}]},"ts":"1680530179268"} 2023-04-03 13:56:19,271 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 13:56:19,274 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 13:56:19,277 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 877 msec 2023-04-03 13:56:19,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 13:56:19,534 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-03 13:56:19,541 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x42d2ba41 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,545 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18660fc4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,548 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,552 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,556 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 13:56:19,559 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:34634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 13:56:19,566 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x33e5a282 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,570 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5be6782, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,571 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x3e0ba002 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,575 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5610151a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,576 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x142c8aea to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,580 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e837e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,581 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x136956fd to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,584 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3167991b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,585 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x1f537ba4 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,588 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6be8c20b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,589 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x1ecabb06 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,592 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@127a87ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,593 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x1439cba9 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,627 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5fe050a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,628 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x655398b8 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,637 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38ebb9f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,637 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x6a7ef79d to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:19,645 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13112432, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:19,650 DEBUG [hconnection-0x4517549a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,655 DEBUG [hconnection-0xb19ab94-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,656 DEBUG [hconnection-0x33489bfb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,656 DEBUG [hconnection-0x28d803f2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,657 DEBUG [hconnection-0x58a1ccf2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,658 DEBUG [hconnection-0x189464fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,658 DEBUG [hconnection-0x3a0d8d3b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,658 DEBUG [hconnection-0x6b777934-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,660 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,667 DEBUG [hconnection-0x107a23fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:19,668 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36230, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,667 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,667 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,669 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,670 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36236, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,670 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36270, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,683 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36266, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:19,688 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:19,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:19,701 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:19,701 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:19,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:19,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:19,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:19,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:19,704 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:19,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:19,706 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,706 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:19,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:19,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:19,710 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:19,710 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:19,711 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:19,718 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:19,718 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:19,719 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:19,719 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:19,721 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-03 13:56:19,736 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:19,737 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:19,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:19,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:19,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:19,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:19,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:19,762 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:19,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:19,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530239873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:19,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530239874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:19,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530239877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:19,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:19,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530239881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530239877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:19,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530179788, max timestamp=1680530179862Num uniques 859; 2023-04-03 13:56:19,895 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530179788, max timestamp=1680530179808Num uniques 859; 2023-04-03 13:56:19,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 13:56:19,905 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 13:56:19,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530179788, max timestamp=1680530179862Num uniques 859; 2023-04-03 13:56:19,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 13:56:20,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530180055, max timestamp=1680530180077Num uniques 1009; 2023-04-03 13:56:20,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-03 13:56:20,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530180055, max timestamp=1680530180076Num uniques 1009; 2023-04-03 13:56:20,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-03 13:56:20,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7681, totalHeapSize=17712, min timestamp=1680530180055, max timestamp=1680530180077Num uniques 1009; 2023-04-03 13:56:20,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=7681, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-03 13:56:20,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530180127, max timestamp=1680530180132Num uniques 1027; 2023-04-03 13:56:20,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680530180117, max timestamp=1680530180132Num uniques 1027; 2023-04-03 13:56:20,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530180127, max timestamp=1680530180132Num uniques 1009; 2023-04-03 13:56:20,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-03 13:56:20,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-03 13:56:20,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-03 13:56:20,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530180171, max timestamp=1680530180180Num uniques 1009; 2023-04-03 13:56:20,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-03 13:56:20,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680530180167, max timestamp=1680530180180Num uniques 1009; 2023-04-03 13:56:20,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10806, suffix heap size=132904,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=236 2023-04-03 13:56:20,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530180171, max timestamp=1680530180180Num uniques 1086; 2023-04-03 13:56:20,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-03 13:56:20,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680530180217, max timestamp=1680530180227Num uniques 1045; 2023-04-03 13:56:20,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-03 13:56:20,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680530180217, max timestamp=1680530180227Num uniques 1095; 2023-04-03 13:56:20,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-03 13:56:20,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680530180217, max timestamp=1680530180227Num uniques 1045; 2023-04-03 13:56:20,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-03 13:56:20,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680530180257, max timestamp=1680530180285Num uniques 1054; 2023-04-03 13:56:20,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680530180257, max timestamp=1680530180285Num uniques 1054; 2023-04-03 13:56:20,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-03 13:56:20,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680530180257, max timestamp=1680530180285Num uniques 1054; 2023-04-03 13:56:20,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-03 13:56:20,289 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-03 13:56:20,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680530180308, max timestamp=1680530180320Num uniques 1063; 2023-04-03 13:56:20,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-03 13:56:20,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680530180308, max timestamp=1680530180320Num uniques 1063; 2023-04-03 13:56:20,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-03 13:56:20,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680530180308, max timestamp=1680530180320Num uniques 1063; 2023-04-03 13:56:20,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-03 13:56:20,358 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/b3ece2e8ce12486897d8132ba5fa8b7f 2023-04-03 13:56:20,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680530180339, max timestamp=1680530180417Num uniques 1022; 2023-04-03 13:56:20,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530180339, max timestamp=1680530180417Num uniques 1022; 2023-04-03 13:56:20,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530180339, max timestamp=1680530180417Num uniques 1022; 2023-04-03 13:56:20,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=6870, suffix heap size=134256,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=150 2023-04-03 13:56:20,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-03 13:56:20,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=6870, suffix heap size=134256,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=150 2023-04-03 13:56:20,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10571, totalHeapSize=24264, min timestamp=1680530180455, max timestamp=1680530180461Num uniques 1009; 2023-04-03 13:56:20,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46209, new segment data size=10571, suffix heap size=132904,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=231 2023-04-03 13:56:20,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10566, totalHeapSize=24264, min timestamp=1680530180455, max timestamp=1680530180461Num uniques 1031; 2023-04-03 13:56:20,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47206, new segment data size=10566, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-03 13:56:20,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10566, totalHeapSize=24264, min timestamp=1680530180455, max timestamp=1680530180461Num uniques 1009; 2023-04-03 13:56:20,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=10566, suffix heap size=132904,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=231 2023-04-03 13:56:20,508 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/188aca54624743e5997e1b60bd15441f 2023-04-03 13:56:20,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680530180501, max timestamp=1680530180506Num uniques 1090; 2023-04-03 13:56:20,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-03 13:56:20,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680530180501, max timestamp=1680530180508Num uniques 1090; 2023-04-03 13:56:20,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-03 13:56:20,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680530180501, max timestamp=1680530180508Num uniques 1090; 2023-04-03 13:56:20,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-03 13:56:20,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680530180527, max timestamp=1680530180548Num uniques 1049; 2023-04-03 13:56:20,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680530180527, max timestamp=1680530180545Num uniques 1049; 2023-04-03 13:56:20,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-03 13:56:20,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680530180527, max timestamp=1680530180545Num uniques 1049; 2023-04-03 13:56:20,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-03 13:56:20,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-04-03 13:56:20,578 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/d8ee97500dcb42f18c6c814859a3ea61 2023-04-03 13:56:20,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680530180587, max timestamp=1680530180592Num uniques 1158; 2023-04-03 13:56:20,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-03 13:56:20,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680530180587, max timestamp=1680530180592Num uniques 1158; 2023-04-03 13:56:20,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-03 13:56:20,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680530180587, max timestamp=1680530180592Num uniques 1158; 2023-04-03 13:56:20,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=7230, suffix heap size=148400,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=158 2023-04-03 13:56:20,597 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/b3ece2e8ce12486897d8132ba5fa8b7f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b3ece2e8ce12486897d8132ba5fa8b7f 2023-04-03 13:56:20,615 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b3ece2e8ce12486897d8132ba5fa8b7f, entries=50, sequenceid=7, filesize=7.0 K 2023-04-03 13:56:20,630 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/188aca54624743e5997e1b60bd15441f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/188aca54624743e5997e1b60bd15441f 2023-04-03 13:56:20,654 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/188aca54624743e5997e1b60bd15441f, entries=50, sequenceid=7, filesize=7.0 K 2023-04-03 13:56:20,664 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/d8ee97500dcb42f18c6c814859a3ea61 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d8ee97500dcb42f18c6c814859a3ea61 2023-04-03 13:56:20,688 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d8ee97500dcb42f18c6c814859a3ea61, entries=50, sequenceid=7, filesize=7.0 K 2023-04-03 13:56:20,691 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=120.76 KB/123660 for 3d9e5bf7331338c5e5778539b0a0c908 in 971ms, sequenceid=7, compaction requested=false 2023-04-03 13:56:20,695 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-03 13:56:20,703 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:20,703 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:20,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:20,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:20,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:20,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:20,707 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,707 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:20,708 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:20,708 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:20,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=617, cellSize=28252, totalHeapSize=64408, min timestamp=1680530180600, max timestamp=1680530180696Num uniques 1017; 2023-04-03 13:56:20,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=28252, suffix heap size=133736,new segment heap size=64408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=617 2023-04-03 13:56:20,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=617, cellSize=28252, totalHeapSize=64408, min timestamp=1680530180600, max timestamp=1680530180696Num uniques 1017; 2023-04-03 13:56:20,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=28252, suffix heap size=133736,new segment heap size=64408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=617 2023-04-03 13:56:20,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=617, cellSize=28252, totalHeapSize=64408, min timestamp=1680530180600, max timestamp=1680530180696Num uniques 1017; 2023-04-03 13:56:20,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=28252, suffix heap size=133736,new segment heap size=64408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=617 2023-04-03 13:56:20,714 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f562a2d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:20,714 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:20,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:20,720 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=87.22 KB heapSize=202.59 KB 2023-04-03 13:56:20,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:20,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:20,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:20,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:20,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:20,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:20,725 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:20,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:20,725 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:20,725 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:20,725 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:20,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:20,747 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:20,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:20,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:20,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:20,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:20,759 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:20,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:20,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:20,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:20,759 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:20,777 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:20,778 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:20,778 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:20,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:20,778 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:20,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:20,782 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:20,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:20,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:20,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:20,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:20,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:20,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:20,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:20,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:20,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:20,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:20,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:20,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:20,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:20,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:20,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:20,809 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:20,809 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:20,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/8fdd8032155b45b991e5463301ab4535 2023-04-03 13:56:20,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530240808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530240809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,844 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530240816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530240827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530240825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/bc55fad51a144476b07013f7939ac091 2023-04-03 13:56:20,909 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:20,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:20,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:20,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:20,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:20,954 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:20,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530240948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,954 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:20,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530240949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530240950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530240950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:20,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:20,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:20,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:20,981 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:20,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:20,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:20,983 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:20,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530240968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:21,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:21,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:21,020 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:21,020 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,020 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:21,020 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,039 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:21,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:21,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:21,055 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/0a7ae797162147c48bbc4c1fd06a538f 2023-04-03 13:56:21,061 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,061 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:21,061 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:21,062 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,068 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:21,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:21,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:21,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:21,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/8fdd8032155b45b991e5463301ab4535 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/8fdd8032155b45b991e5463301ab4535 2023-04-03 13:56:21,074 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,074 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,075 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:21,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e307c4c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:21,077 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:21,077 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,079 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:21,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:21,079 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:21,079 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:21,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:21,085 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:21,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/8fdd8032155b45b991e5463301ab4535, entries=600, sequenceid=234, filesize=32.8 K 2023-04-03 13:56:21,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/bc55fad51a144476b07013f7939ac091 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bc55fad51a144476b07013f7939ac091 2023-04-03 13:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:21,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:21,105 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:21,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:21,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:21,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:21,105 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:21,108 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,108 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:21,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,123 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bc55fad51a144476b07013f7939ac091, entries=150, sequenceid=234, filesize=11.8 K 2023-04-03 13:56:21,125 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:21,125 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:21,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:21,126 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:21,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:21,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:21,126 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:21,126 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,126 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:21,126 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,127 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:21,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:21,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/0a7ae797162147c48bbc4c1fd06a538f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/0a7ae797162147c48bbc4c1fd06a538f 2023-04-03 13:56:21,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/0a7ae797162147c48bbc4c1fd06a538f, entries=150, sequenceid=234, filesize=11.8 K 2023-04-03 13:56:21,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~202.55 KB/207408, currentSize=107.34 KB/109920 for 3d9e5bf7331338c5e5778539b0a0c908 in 443ms, sequenceid=234, compaction requested=false 2023-04-03 13:56:21,165 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:21,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:21,183 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 13:56:21,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:21,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:21,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:21,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:21,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:21,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:21,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530241201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530241204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530241205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530241213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,230 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:21,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530241216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:21,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:21,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:21,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:21,278 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:21,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:21,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:21,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/fb4e1acf0a68461d949b10d8789b793b 2023-04-03 13:56:21,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,287 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:21,287 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:21,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:21,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:21,295 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:21,295 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:21,306 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:21,306 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:21,306 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,306 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:21,306 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:21,306 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:21,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:21,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:21,308 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:21,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530241315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530241316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530241316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,323 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,323 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530241318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:21,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7ac06f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:21,326 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:21,326 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530241332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,335 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:21,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:21,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:21,342 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:21,342 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:21,342 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:21,351 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:21,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:21,357 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:21,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:21,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:21,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:21,357 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:21,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,361 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,364 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:21,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:21,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:21,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:21,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:21,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:21,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,392 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:21,392 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:21,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:21,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:21,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:21,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:21,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:21,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:21,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:21,519 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:21,519 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:21,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:21,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,523 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530241522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,523 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:21,527 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:21,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:21,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:21,529 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:21,529 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,531 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:21,532 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:21,532 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:21,532 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,532 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:21,532 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:21,532 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,533 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:21,533 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:21,533 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,533 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:21,533 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:21,541 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530241532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,545 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,545 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,545 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:21,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:21,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44b4cf25[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:21,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:21,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,551 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:21,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:21,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:21,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:21,559 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:21,559 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:21,559 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:21,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,563 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530241548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530241548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530241548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:21,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:21,569 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:21,569 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:21,569 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:21,569 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:21,569 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:21,570 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,571 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:21,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,597 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,597 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:21,597 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:21,597 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,597 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:21,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:21,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:21,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:21,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:21,597 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,599 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,599 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,599 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:21,599 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:21,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:21,699 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:21,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:21,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:21,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:21,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:21,715 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:21,716 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,723 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:21,723 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:21,724 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,724 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:21,724 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:21,725 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,725 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:21,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:21,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:21,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:21,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:21,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:21,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,731 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:21,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:21,734 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:21,734 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:21,734 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:21,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:21,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:21,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:21,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:21,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,740 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:21,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7d82dc[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:21,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:21,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:21,746 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:21,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:21,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:21,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:21,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:21,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:21,770 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:21,770 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:21,770 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:21,770 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:21,770 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:21,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:21,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:21,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:21,773 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,784 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,785 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:21,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:21,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:21,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:21,787 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:21,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:21,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:21,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:21,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:21,787 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:21,787 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:21,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:21,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:21,809 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/e824049a43e94074ad89067bb9de8316 2023-04-03 13:56:21,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530241827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530241849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530241872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,888 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:21,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:21,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/34258749c1a743e2b36adae996de0cce 2023-04-03 13:56:21,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530241899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:21,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:21,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530241900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:21,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:21,911 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:21,911 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,912 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:21,912 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:21,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,918 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/fb4e1acf0a68461d949b10d8789b793b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fb4e1acf0a68461d949b10d8789b793b 2023-04-03 13:56:21,918 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:21,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:21,918 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,919 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:21,919 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:21,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:21,930 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fb4e1acf0a68461d949b10d8789b793b, entries=200, sequenceid=254, filesize=14.1 K 2023-04-03 13:56:21,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/e824049a43e94074ad89067bb9de8316 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e824049a43e94074ad89067bb9de8316 2023-04-03 13:56:21,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:21,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:21,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:21,962 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:21,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:21,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:21,975 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,975 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:21,975 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:21,975 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:21,988 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e824049a43e94074ad89067bb9de8316, entries=150, sequenceid=254, filesize=11.8 K 2023-04-03 13:56:21,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/34258749c1a743e2b36adae996de0cce as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/34258749c1a743e2b36adae996de0cce 2023-04-03 13:56:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,003 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:22,003 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:22,005 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:22,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f25dc3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:22,022 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:22,023 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:22,024 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:22,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:22,024 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:22,024 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,024 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:22,024 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:22,024 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:22,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:22,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:22,026 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:22,027 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:22,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:22,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:22,032 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:22,032 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:22,032 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:22,032 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:22,033 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:22,035 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,035 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:22,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,035 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:22,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:22,036 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:22,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/34258749c1a743e2b36adae996de0cce, entries=150, sequenceid=254, filesize=11.8 K 2023-04-03 13:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:22,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:22,041 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3d9e5bf7331338c5e5778539b0a0c908 in 857ms, sequenceid=254, compaction requested=true 2023-04-03 13:56:22,041 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:22,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:22,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:22,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:22,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:22,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:22,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:22,051 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:22,051 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:22,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:22,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:22,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:22,052 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:22,052 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:22,054 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:22,054 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:22,054 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:22,056 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:22,057 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:22,070 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:22,072 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 55176 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:22,073 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/A is initiating minor compaction (all files) 2023-04-03 13:56:22,073 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/A in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:22,073 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/B is initiating minor compaction (all files) 2023-04-03 13:56:22,073 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/B in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:22,073 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b3ece2e8ce12486897d8132ba5fa8b7f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/8fdd8032155b45b991e5463301ab4535, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fb4e1acf0a68461d949b10d8789b793b] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=53.9 K 2023-04-03 13:56:22,073 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/188aca54624743e5997e1b60bd15441f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bc55fad51a144476b07013f7939ac091, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e824049a43e94074ad89067bb9de8316] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=30.5 K 2023-04-03 13:56:22,076 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 188aca54624743e5997e1b60bd15441f, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1680530179692 2023-04-03 13:56:22,076 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting b3ece2e8ce12486897d8132ba5fa8b7f, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1680530179692 2023-04-03 13:56:22,081 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 8fdd8032155b45b991e5463301ab4535, keycount=600, bloomtype=ROW, size=32.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680530180600 2023-04-03 13:56:22,086 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bc55fad51a144476b07013f7939ac091, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680530180652 2023-04-03 13:56:22,086 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting fb4e1acf0a68461d949b10d8789b793b, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530180783 2023-04-03 13:56:22,094 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e824049a43e94074ad89067bb9de8316, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530180804 2023-04-03 13:56:22,137 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#A#compaction#9 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:22,153 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:22,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:22,162 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#B#compaction#10 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:22,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:22,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:22,175 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:22,176 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:22,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:22,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:22,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,188 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:22,189 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:22,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:22,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:22,191 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,191 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:22,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:22,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:22,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:22,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:22,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:22,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:22,223 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:22,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:22,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:22,239 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:22,243 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:22,243 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:56:22,246 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:22,246 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:22,247 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:22,247 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:22,247 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:22,247 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:22,286 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/f0dd05b5651148b68464ee5c6b7ee921 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f0dd05b5651148b68464ee5c6b7ee921 2023-04-03 13:56:22,287 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/d84d9969421e4bf4b54337831c8b3d5f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d84d9969421e4bf4b54337831c8b3d5f 2023-04-03 13:56:22,345 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/B of 3d9e5bf7331338c5e5778539b0a0c908 into d84d9969421e4bf4b54337831c8b3d5f(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:22,345 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/A of 3d9e5bf7331338c5e5778539b0a0c908 into f0dd05b5651148b68464ee5c6b7ee921(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:22,345 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:22,347 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:22,347 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/A, priority=13, startTime=1680530182042; duration=0sec 2023-04-03 13:56:22,347 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/B, priority=13, startTime=1680530182054; duration=0sec 2023-04-03 13:56:22,349 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:22,350 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:22,353 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:22,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:22,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:22,360 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 31272 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:22,360 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/C is initiating minor compaction (all files) 2023-04-03 13:56:22,360 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/C in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:22,360 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d8ee97500dcb42f18c6c814859a3ea61, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/0a7ae797162147c48bbc4c1fd06a538f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/34258749c1a743e2b36adae996de0cce] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=30.5 K 2023-04-03 13:56:22,361 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/c9b1ef0ff61c48be9459c844e756a8f5 2023-04-03 13:56:22,361 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting d8ee97500dcb42f18c6c814859a3ea61, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1680530179692 2023-04-03 13:56:22,365 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 0a7ae797162147c48bbc4c1fd06a538f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1680530180652 2023-04-03 13:56:22,366 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 34258749c1a743e2b36adae996de0cce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530180804 2023-04-03 13:56:22,399 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#C#compaction#13 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:22,430 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/53e9362625b24d2d9744645218539d7a 2023-04-03 13:56:22,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530242435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530242430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530242449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530242450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530242449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,482 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/33eb10cd1905439b8a3b296c5293837c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/33eb10cd1905439b8a3b296c5293837c 2023-04-03 13:56:22,500 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/C of 3d9e5bf7331338c5e5778539b0a0c908 into 33eb10cd1905439b8a3b296c5293837c(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:22,500 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:22,501 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/C, priority=13, startTime=1680530182054; duration=0sec 2023-04-03 13:56:22,501 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:22,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530242552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530242564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530242566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530242574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530242576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,620 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 13:56:22,770 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-03 13:56:22,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530242769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,772 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-03 13:56:22,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530242774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530242780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:22,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530242780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530242780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:22,950 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=266 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/395aa8afcb554baf9c449c349c6b1f73 2023-04-03 13:56:22,982 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/c9b1ef0ff61c48be9459c844e756a8f5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/c9b1ef0ff61c48be9459c844e756a8f5 2023-04-03 13:56:23,003 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/c9b1ef0ff61c48be9459c844e756a8f5, entries=150, sequenceid=266, filesize=11.9 K 2023-04-03 13:56:23,008 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/53e9362625b24d2d9744645218539d7a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/53e9362625b24d2d9744645218539d7a 2023-04-03 13:56:23,023 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/53e9362625b24d2d9744645218539d7a, entries=150, sequenceid=266, filesize=11.9 K 2023-04-03 13:56:23,032 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/395aa8afcb554baf9c449c349c6b1f73 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/395aa8afcb554baf9c449c349c6b1f73 2023-04-03 13:56:23,063 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/395aa8afcb554baf9c449c349c6b1f73, entries=150, sequenceid=266, filesize=11.9 K 2023-04-03 13:56:23,083 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3d9e5bf7331338c5e5778539b0a0c908 in 824ms, sequenceid=266, compaction requested=false 2023-04-03 13:56:23,083 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:23,083 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,083 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:23,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:23,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:23,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:23,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,092 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,092 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:23,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530182406, max timestamp=1680530183093Num uniques 859; 2023-04-03 13:56:23,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:23,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530182406, max timestamp=1680530183093Num uniques 859; 2023-04-03 13:56:23,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:23,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:23,097 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-03 13:56:23,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:23,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:23,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:23,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:23,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:23,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:23,098 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6286ea[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:23,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:23,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,110 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,110 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:23,110 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:23,110 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:23,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:23,112 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:23,112 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,112 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:23,113 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:23,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:23,118 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:23,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:23,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:23,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:23,118 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:23,126 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,126 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,129 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,142 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530243145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530183142, max timestamp=1680530183151Num uniques 859; 2023-04-03 13:56:23,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:23,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530183142, max timestamp=1680530183151Num uniques 859; 2023-04-03 13:56:23,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530183142, max timestamp=1680530183151Num uniques 859; 2023-04-03 13:56:23,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:23,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:23,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:23,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:23,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,163 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,163 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:23,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:23,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,163 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,164 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,164 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:23,164 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:23,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530243182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530243184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530243184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530243189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530243255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,264 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:23,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:23,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:23,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:23,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:23,283 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:23,283 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:23,287 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:23,287 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:23,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,291 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:23,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:23,292 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,292 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,292 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:23,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530243292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,297 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530243293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530243299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530243299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:23,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:23,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:23,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:23,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:23,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:23,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:23,311 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,312 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:23,312 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:23,312 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:23,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:23,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:23,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:23,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:23,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4dabe4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:23,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:23,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,325 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:23,325 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:23,326 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,326 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:23,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:23,335 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:23,349 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:23,349 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:23,349 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:23,349 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:23,349 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:23,351 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,351 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,351 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,351 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,353 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,359 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:23,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:23,361 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:23,361 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:23,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,461 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:23,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:23,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:23,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530243467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:23,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:23,482 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:23,483 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:23,491 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:23,491 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:23,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,493 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:23,493 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:23,494 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,494 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:23,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:23,496 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:23,496 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:23,497 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:23,500 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:23,500 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,500 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:23,500 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:23,500 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,501 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:23,501 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:23,501 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,501 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:23,501 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:23,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530243503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530243507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530243508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,516 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:23,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a40b9d3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:23,519 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:23,519 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,520 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,520 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:23,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530243503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:23,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:23,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:23,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:23,524 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:23,524 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,524 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,525 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:23,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:23,529 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:23,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:23,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:23,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:23,529 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:23,530 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,530 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,531 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,533 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,540 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,545 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,554 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,554 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:23,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:23,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:23,555 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:23,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/48c05eeebc9d444d9497b817074021c0 2023-04-03 13:56:23,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:23,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:23,555 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:23,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/05b661bbc44c4bd8b833f5be80862759 2023-04-03 13:56:23,656 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:23,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:23,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:23,670 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:23,670 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:23,671 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:23,671 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:23,672 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,672 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,673 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,673 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:23,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,676 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:23,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/45eb3e8939a8464583ff378842be6ea4 2023-04-03 13:56:23,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:23,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:23,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:23,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:23,680 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:23,680 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:23,682 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:23,692 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:23,692 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:23,704 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,704 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:23,704 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:23,705 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:23,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:23,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:23,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:23,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/48c05eeebc9d444d9497b817074021c0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/48c05eeebc9d444d9497b817074021c0 2023-04-03 13:56:23,707 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,707 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,708 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,708 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:23,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f19f71d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:23,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:23,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,712 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,712 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,712 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:23,712 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:23,712 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:23,712 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:23,715 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,716 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:23,716 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,716 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,727 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/48c05eeebc9d444d9497b817074021c0, entries=200, sequenceid=290, filesize=14.3 K 2023-04-03 13:56:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/05b661bbc44c4bd8b833f5be80862759 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/05b661bbc44c4bd8b833f5be80862759 2023-04-03 13:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:23,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:23,735 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:23,735 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:23,735 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:23,735 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:23,735 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:23,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,748 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/05b661bbc44c4bd8b833f5be80862759, entries=150, sequenceid=290, filesize=11.9 K 2023-04-03 13:56:23,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,767 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,780 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/45eb3e8939a8464583ff378842be6ea4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/45eb3e8939a8464583ff378842be6ea4 2023-04-03 13:56:23,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:23,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:23,782 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,782 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,782 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:23,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:23,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:23,782 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,782 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,783 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,783 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,783 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,783 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:23,783 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,783 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:23,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530243786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/45eb3e8939a8464583ff378842be6ea4, entries=150, sequenceid=290, filesize=11.9 K 2023-04-03 13:56:23,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~58.14 KB/59540, heapSize ~164.13 KB/168064, currentSize=127.47 KB/130530 for 3d9e5bf7331338c5e5778539b0a0c908 in 711ms, sequenceid=290, compaction requested=true 2023-04-03 13:56:23,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:23,808 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:23,811 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:23,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:23,822 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:23,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:23,824 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38951 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:23,825 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/A is initiating minor compaction (all files) 2023-04-03 13:56:23,825 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36511 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:23,825 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/B is initiating minor compaction (all files) 2023-04-03 13:56:23,825 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/B in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,825 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d84d9969421e4bf4b54337831c8b3d5f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/53e9362625b24d2d9744645218539d7a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/05b661bbc44c4bd8b833f5be80862759] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=35.7 K 2023-04-03 13:56:23,825 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/A in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,826 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f0dd05b5651148b68464ee5c6b7ee921, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/c9b1ef0ff61c48be9459c844e756a8f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/48c05eeebc9d444d9497b817074021c0] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=38.0 K 2023-04-03 13:56:23,826 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d84d9969421e4bf4b54337831c8b3d5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530180804 2023-04-03 13:56:23,828 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f0dd05b5651148b68464ee5c6b7ee921, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530180804 2023-04-03 13:56:23,830 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 53e9362625b24d2d9744645218539d7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680530181202 2023-04-03 13:56:23,831 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting c9b1ef0ff61c48be9459c844e756a8f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680530181202 2023-04-03 13:56:23,832 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 05b661bbc44c4bd8b833f5be80862759, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680530182406 2023-04-03 13:56:23,833 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 48c05eeebc9d444d9497b817074021c0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680530182406 2023-04-03 13:56:23,837 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 13:56:23,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:23,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:23,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:23,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:23,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:23,839 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:23,839 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:23,866 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#B#compaction#19 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:23,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530243857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,898 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:23,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530243862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530243863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530243864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:23,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:23,909 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#A#compaction#20 average throughput is 0.25 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:23,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:23,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:23,930 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:23,930 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:23,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:23,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:23,946 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,946 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,949 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:23,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:23,950 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:23,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:23,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:23,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:23,953 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:23,953 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:23,958 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:23,958 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:23,958 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:23,961 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,961 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:23,961 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:23,962 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:23,962 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:23,962 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:23,962 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,962 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:23,962 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:23,964 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,966 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,966 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:23,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55666bc2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:23,971 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:23,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:23,979 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:23,979 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:23,979 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:23,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:23,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:23,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:23,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:23,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:23,985 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:23,985 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:23,985 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:23,985 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:23,985 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:23,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:23,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530243983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:23,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:23,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:23,992 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,994 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:23,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:24,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:24,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:24,002 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:24,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:24,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530244005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530244005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530244005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,075 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-03 13:56:24,075 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-03 13:56:24,102 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:24,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:24,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:24,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:24,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:24,107 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:24,107 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,108 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:24,108 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:24,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,111 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:24,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:24,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:24,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:24,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:24,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,115 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:24,115 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:24,115 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:24,115 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,115 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:24,115 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:24,115 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,115 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:24,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:24,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:24,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:24,117 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:24,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:24,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c5e801[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:24,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:24,122 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:24,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:24,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:24,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:24,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:24,133 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:24,133 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:24,133 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:24,133 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:24,133 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:24,142 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,142 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,146 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,147 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,148 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,152 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,152 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,152 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,152 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:24,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:24,152 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,152 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,152 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,154 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,153 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:24,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:24,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530244187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530244209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530244210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530244212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:24,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:24,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:24,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:24,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:24,259 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:24,259 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,260 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:24,260 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:24,261 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,261 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,262 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,262 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,262 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:24,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:24,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,264 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:24,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:24,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:24,265 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:24,265 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,265 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:24,266 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:24,266 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:24,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:24,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:24,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,266 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:24,266 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:24,266 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,266 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:24,266 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:24,268 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,268 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,268 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,268 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:24,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bf44b6b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:24,270 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:24,270 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,271 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,271 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:24,271 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,271 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:24,271 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:24,271 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:24,272 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:24,272 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,272 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:24,272 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:24,277 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:24,277 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:24,277 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:24,277 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:24,277 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:24,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,287 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:24,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:24,293 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,294 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,299 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,299 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,299 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,299 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,299 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:24,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:24,299 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:24,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530244300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,362 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/50f3ed60b6b74353b1ab97b24f0ddd46 2023-04-03 13:56:24,374 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/81d8176bc10246bfaaf50c87eb2364db as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/81d8176bc10246bfaaf50c87eb2364db 2023-04-03 13:56:24,376 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/51c1394cd13141afac8803b091272de0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/51c1394cd13141afac8803b091272de0 2023-04-03 13:56:24,397 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/B of 3d9e5bf7331338c5e5778539b0a0c908 into 81d8176bc10246bfaaf50c87eb2364db(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:24,398 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:24,399 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/B, priority=13, startTime=1680530183822; duration=0sec 2023-04-03 13:56:24,408 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:24,411 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:24,411 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:24,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/2c5ac10808e64df49b1042e67158a09a 2023-04-03 13:56:24,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:24,413 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/A of 3d9e5bf7331338c5e5778539b0a0c908 into 51c1394cd13141afac8803b091272de0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:24,413 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:24,413 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/A, priority=13, startTime=1680530183808; duration=0sec 2023-04-03 13:56:24,413 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36511 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:24,413 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:24,413 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/C is initiating minor compaction (all files) 2023-04-03 13:56:24,413 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/C in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,413 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/33eb10cd1905439b8a3b296c5293837c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/395aa8afcb554baf9c449c349c6b1f73, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/45eb3e8939a8464583ff378842be6ea4] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=35.7 K 2023-04-03 13:56:24,414 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 33eb10cd1905439b8a3b296c5293837c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530180804 2023-04-03 13:56:24,415 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 395aa8afcb554baf9c449c349c6b1f73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=266, earliestPutTs=1680530181202 2023-04-03 13:56:24,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:24,415 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 45eb3e8939a8464583ff378842be6ea4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680530182406 2023-04-03 13:56:24,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:24,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:24,419 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:24,419 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,419 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:24,419 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:24,420 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,420 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,421 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,421 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,422 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,422 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:24,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:24,424 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:24,424 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,425 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:24,425 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:24,425 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:24,425 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,425 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:24,425 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:24,426 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:24,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:24,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:24,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:24,436 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,437 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,437 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,438 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#C#compaction#23 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:24,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b65ee17[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:24,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:24,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:24,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:24,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:24,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:24,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:24,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:24,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:24,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:24,451 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:24,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:24,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:24,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:24,451 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:24,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:24,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,477 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,478 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:24,478 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:24,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,493 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530244492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530244512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530244516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530244521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,579 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:24,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:24,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:24,602 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:24,602 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:24,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:24,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,605 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,606 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,607 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,607 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:24,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:24,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:24,607 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:24,607 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,608 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:24,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:24,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:24,609 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,609 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:24,609 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:24,609 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:24,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:24,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:24,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:24,619 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,620 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:24,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:24,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dbbb0c2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:24,628 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:24,629 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:24,638 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:24,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:24,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:24,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:24,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:24,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:24,649 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:24,650 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:24,650 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:24,650 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:24,650 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:24,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,655 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,658 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,658 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:24,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:24,658 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,659 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,659 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,659 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,659 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,659 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,660 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:24,660 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:24,662 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,662 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:24,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,760 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:24,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:24,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:24,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:24,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:24,772 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:24,772 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,773 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:24,773 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:24,774 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,774 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,775 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:24,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:24,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:24,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:24,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:24,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:24,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:24,785 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:24,785 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:24,786 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,797 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:24,797 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:24,797 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:24,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:24,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:24,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:24,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:24,800 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:24,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4675d494[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:24,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:24,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:24,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:24,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:24,811 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,811 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:24,818 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:24,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:24,836 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:24,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:24,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:24,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:24,836 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:24,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:24,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:24,843 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:24,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,850 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,854 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,854 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,854 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,854 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:24,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:24,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:24,854 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,855 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,855 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:24,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:24,855 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:24,860 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,859 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:24,860 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:24,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/f938c1d771154f9989c86768efa612a5 2023-04-03 13:56:24,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/50f3ed60b6b74353b1ab97b24f0ddd46 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/50f3ed60b6b74353b1ab97b24f0ddd46 2023-04-03 13:56:24,881 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/e677ecd730b0422eaa72c7ac5e4d96fc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e677ecd730b0422eaa72c7ac5e4d96fc 2023-04-03 13:56:24,892 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/50f3ed60b6b74353b1ab97b24f0ddd46, entries=250, sequenceid=327, filesize=16.7 K 2023-04-03 13:56:24,895 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/C of 3d9e5bf7331338c5e5778539b0a0c908 into e677ecd730b0422eaa72c7ac5e4d96fc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:24,903 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:24,903 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/C, priority=13, startTime=1680530183822; duration=0sec 2023-04-03 13:56:24,903 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:24,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/2c5ac10808e64df49b1042e67158a09a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c5ac10808e64df49b1042e67158a09a 2023-04-03 13:56:24,921 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c5ac10808e64df49b1042e67158a09a, entries=150, sequenceid=327, filesize=11.9 K 2023-04-03 13:56:24,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/f938c1d771154f9989c86768efa612a5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/f938c1d771154f9989c86768efa612a5 2023-04-03 13:56:24,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/f938c1d771154f9989c86768efa612a5, entries=150, sequenceid=327, filesize=11.9 K 2023-04-03 13:56:24,943 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 3d9e5bf7331338c5e5778539b0a0c908 in 1106ms, sequenceid=327, compaction requested=false 2023-04-03 13:56:24,943 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:24,960 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:24,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:24,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:24,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:24,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:24,982 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:24,982 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:24,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:24,984 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,984 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,985 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:24,985 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:24,985 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:24,985 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:24,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:24,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:24,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:24,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:24,996 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:24,996 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:24,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:25,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:25,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:56:25,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:25,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:25,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:25,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:25,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:25,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:25,009 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:25,009 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:25,009 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,009 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:25,009 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:25,009 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,018 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:25,018 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:25,018 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,018 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:25,019 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:25,020 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,020 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,021 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,027 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:25,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa36022[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:25,031 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:25,031 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,035 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:25,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:25,035 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,035 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:25,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:25,036 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,036 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:25,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:25,039 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:25,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:25,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:25,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:25,040 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:25,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,042 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:25,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:25,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:25,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,104 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/7b8b9ea48b9d4a87a94e87da1d138880 2023-04-03 13:56:25,148 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:25,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680530185094, max timestamp=1680530185144Num uniques 859; 2023-04-03 13:56:25,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:25,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:25,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530185094, max timestamp=1680530185144Num uniques 859; 2023-04-03 13:56:25,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530185094, max timestamp=1680530185144Num uniques 859; 2023-04-03 13:56:25,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:25,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:25,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:25,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:25,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:25,182 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:25,182 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:25,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:25,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,211 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,212 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,212 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,212 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:25,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:25,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530245201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530245208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:25,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:25,224 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:25,224 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:25,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:25,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:25,226 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,226 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:25,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530245198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,226 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:25,242 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:25,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530245223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:25,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:25,244 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,244 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,246 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b722497[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:25,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:25,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,264 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,264 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:25,264 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:25,264 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:25,264 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:25,267 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,268 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:25,268 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,268 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:25,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:25,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:25,276 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:25,276 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:25,276 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:25,276 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:25,276 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:25,278 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,278 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,278 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,281 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,283 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:25,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:25,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:25,301 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:25,301 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:25,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530245308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,320 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530245318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530245320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530245329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530245346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,401 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:25,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:25,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:25,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:25,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:25,411 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:25,411 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,417 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:25,417 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:25,418 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,418 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,419 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,419 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,420 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,420 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:25,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:25,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:25,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:25,424 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,425 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:25,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:25,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:25,426 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,426 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:25,426 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:25,426 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:25,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:25,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:25,426 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:25,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,435 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b9fa69b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:25,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:25,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:25,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:25,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:25,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:25,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:25,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:25,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:25,450 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:25,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:25,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:25,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:25,450 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:25,458 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,458 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,458 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,460 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:25,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:25,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,471 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:25,471 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:25,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530245523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530245525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530245534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530245549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,571 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:25,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:25,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:25,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:25,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:25,586 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:25,586 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,588 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:25,588 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:25,589 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,589 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,590 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,591 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:25,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:25,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:25,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:25,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,596 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:25,602 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:25,602 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,602 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:25,602 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:25,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:25,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:25,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:25,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:25,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:25,606 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,606 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,606 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:25,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/ec8b856bc98a4197a99dc7b3527574b8 2023-04-03 13:56:25,614 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:25,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43a60608[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:25,620 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:25,620 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,622 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:25,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:25,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:25,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:25,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:25,624 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,630 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:25,630 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:25,630 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:25,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:25,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:25,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:25,631 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:25,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,635 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:25,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:25,642 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:25,642 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:25,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:25,742 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:25,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:25,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:25,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:25,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:25,754 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:25,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:25,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:25,755 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,755 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:25,756 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:25,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:25,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:25,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:25,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:25,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:25,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:25,778 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,778 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:25,779 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:25,779 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:25,779 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:25,779 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:25,779 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,779 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:25,779 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:25,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,786 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,786 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,786 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:25,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6720cc2e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:25,791 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:25,791 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,793 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,794 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:25,795 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:25,795 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:25,795 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,795 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:25,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:25,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,809 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:25,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:25,820 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:25,820 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:25,820 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:25,820 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:25,820 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:25,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,822 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,831 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:25,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,836 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:25,837 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:25,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:25,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:25,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530245830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530245830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530245840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:25,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530245859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:25,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:25,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:25,958 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:25,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:25,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:25,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,960 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:25,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:25,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:25,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:25,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:25,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:25,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:25,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:25,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:25,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:25,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:25,976 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,976 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:25,976 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:25,976 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:25,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:25,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:25,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:25,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:25,983 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,983 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,983 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:25,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d1c1ac1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:25,992 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:25,992 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,993 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:25,993 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:25,993 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:25,993 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:25,994 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,994 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:25,995 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:25,995 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,995 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:25,995 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:25,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:25,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:25,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:25,998 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:25,998 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:25,998 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:25,998 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:25,998 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:26,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:26,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:26,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:26,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:26,010 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:26,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:26,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:26,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:26,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:26,017 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:26,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:26,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:26,018 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:26,018 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:26,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:26,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:26,092 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/de8c4d40ce6d49069fd4a0329c3f8541 2023-04-03 13:56:26,106 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/7b8b9ea48b9d4a87a94e87da1d138880 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/7b8b9ea48b9d4a87a94e87da1d138880 2023-04-03 13:56:26,118 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:26,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:26,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/7b8b9ea48b9d4a87a94e87da1d138880, entries=200, sequenceid=340, filesize=14.3 K 2023-04-03 13:56:26,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/ec8b856bc98a4197a99dc7b3527574b8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ec8b856bc98a4197a99dc7b3527574b8 2023-04-03 13:56:26,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:26,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:26,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:26,128 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:26,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:26,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:26,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:26,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:26,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:26,130 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,130 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,130 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:26,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:26,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:26,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:26,131 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:26,131 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:26,131 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:26,134 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:26,134 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:26,134 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:26,134 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:26,134 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:26,134 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:26,135 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:26,135 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:26,135 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,135 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:26,135 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:26,136 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ec8b856bc98a4197a99dc7b3527574b8, entries=150, sequenceid=340, filesize=11.9 K 2023-04-03 13:56:26,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/de8c4d40ce6d49069fd4a0329c3f8541 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/de8c4d40ce6d49069fd4a0329c3f8541 2023-04-03 13:56:26,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,144 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:26,145 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,145 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:26,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:26,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:26,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b61de87[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:26,148 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:26,148 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,151 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,152 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:26,152 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:26,152 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:26,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:26,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:26,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:26,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:26,153 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:26,153 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/de8c4d40ce6d49069fd4a0329c3f8541, entries=150, sequenceid=340, filesize=11.9 K 2023-04-03 13:56:26,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:26,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 3d9e5bf7331338c5e5778539b0a0c908 in 1153ms, sequenceid=340, compaction requested=true 2023-04-03 13:56:26,155 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,155 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:26,156 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:26,156 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:26,156 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:26,156 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:26,158 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:26,158 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/B is initiating minor compaction (all files) 2023-04-03 13:56:26,158 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/B in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:26,158 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44084 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:26,158 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/A is initiating minor compaction (all files) 2023-04-03 13:56:26,158 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/A in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:26,158 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/51c1394cd13141afac8803b091272de0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/50f3ed60b6b74353b1ab97b24f0ddd46, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/7b8b9ea48b9d4a87a94e87da1d138880] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=43.1 K 2023-04-03 13:56:26,158 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/81d8176bc10246bfaaf50c87eb2364db, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c5ac10808e64df49b1042e67158a09a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ec8b856bc98a4197a99dc7b3527574b8] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=35.9 K 2023-04-03 13:56:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,160 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 51c1394cd13141afac8803b091272de0, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680530182406 2023-04-03 13:56:26,161 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 81d8176bc10246bfaaf50c87eb2364db, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680530182406 2023-04-03 13:56:26,161 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 50f3ed60b6b74353b1ab97b24f0ddd46, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680530183181 2023-04-03 13:56:26,162 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2c5ac10808e64df49b1042e67158a09a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680530183181 2023-04-03 13:56:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,162 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 7b8b9ea48b9d4a87a94e87da1d138880, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680530183841 2023-04-03 13:56:26,162 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ec8b856bc98a4197a99dc7b3527574b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680530183841 2023-04-03 13:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:26,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:26,163 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:26,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:26,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:26,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:26,163 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:26,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:26,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:26,175 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:26,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:26,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:26,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:26,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:26,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:26,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,183 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:26,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:26,183 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:26,189 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#A#compaction#27 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:26,196 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#B#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:26,262 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/023e2ac38d834b7d9b6d42bc5add964d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/023e2ac38d834b7d9b6d42bc5add964d 2023-04-03 13:56:26,264 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/2c2fd3021d7a4bbbbb8ab092d72c9556 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c2fd3021d7a4bbbbb8ab092d72c9556 2023-04-03 13:56:26,276 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/A of 3d9e5bf7331338c5e5778539b0a0c908 into 023e2ac38d834b7d9b6d42bc5add964d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:26,279 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:26,279 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/A, priority=13, startTime=1680530186155; duration=0sec 2023-04-03 13:56:26,279 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:26,279 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:26,282 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/B of 3d9e5bf7331338c5e5778539b0a0c908 into 2c2fd3021d7a4bbbbb8ab092d72c9556(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:26,283 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:26,282 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:26,291 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/C is initiating minor compaction (all files) 2023-04-03 13:56:26,292 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:26,292 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/C in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:26,292 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/B, priority=13, startTime=1680530186156; duration=0sec 2023-04-03 13:56:26,292 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e677ecd730b0422eaa72c7ac5e4d96fc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/f938c1d771154f9989c86768efa612a5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/de8c4d40ce6d49069fd4a0329c3f8541] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=35.9 K 2023-04-03 13:56:26,292 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:26,294 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting e677ecd730b0422eaa72c7ac5e4d96fc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680530182406 2023-04-03 13:56:26,295 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f938c1d771154f9989c86768efa612a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680530183181 2023-04-03 13:56:26,295 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting de8c4d40ce6d49069fd4a0329c3f8541, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680530183841 2023-04-03 13:56:26,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:26,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:26,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:26,302 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:26,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:26,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:26,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:26,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,304 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:26,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:26,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:26,316 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#C#compaction#29 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:26,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:26,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:26,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:26,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:26,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:26,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:26,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:26,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:26,345 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:26,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:26,349 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:26,349 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=147.60 KB heapSize=420.84 KB 2023-04-03 13:56:26,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530185199, max timestamp=1680530186343Num uniques 1068; 2023-04-03 13:56:26,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-03 13:56:26,349 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:26,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530185199, max timestamp=1680530186343Num uniques 1068; 2023-04-03 13:56:26,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:26,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-03 13:56:26,350 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:26,350 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:26,350 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:56:26,350 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:26,350 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:26,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530246416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530246417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530246418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530246418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,531 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36270 deadline: 1680530246530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,542 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530246539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,547 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530246544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530246546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530246735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36280 deadline: 1680530246744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36252 deadline: 1680530246751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:26,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36234 deadline: 1680530246752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,793 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/24177b567d1f4bce8477203eb49c3422 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/24177b567d1f4bce8477203eb49c3422 2023-04-03 13:56:26,809 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/C of 3d9e5bf7331338c5e5778539b0a0c908 into 24177b567d1f4bce8477203eb49c3422(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:26,809 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:26,809 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/C, priority=13, startTime=1680530186156; duration=0sec 2023-04-03 13:56:26,809 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:26,839 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/5d833c246320494f99e4e4a4add013e4 2023-04-03 13:56:26,885 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/bdd9b9748a8b4235a483f2878a1c001c 2023-04-03 13:56:26,920 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/6f408cd02f394852ab6642895d718d9b 2023-04-03 13:56:26,939 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/5d833c246320494f99e4e4a4add013e4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/5d833c246320494f99e4e4a4add013e4 2023-04-03 13:56:26,949 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/5d833c246320494f99e4e4a4add013e4, entries=200, sequenceid=380, filesize=14.3 K 2023-04-03 13:56:26,955 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/bdd9b9748a8b4235a483f2878a1c001c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bdd9b9748a8b4235a483f2878a1c001c 2023-04-03 13:56:26,962 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bdd9b9748a8b4235a483f2878a1c001c, entries=150, sequenceid=380, filesize=11.9 K 2023-04-03 13:56:26,967 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/6f408cd02f394852ab6642895d718d9b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/6f408cd02f394852ab6642895d718d9b 2023-04-03 13:56:26,977 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/6f408cd02f394852ab6642895d718d9b, entries=150, sequenceid=380, filesize=11.9 K 2023-04-03 13:56:26,981 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~183.83 KB/188240, currentSize=114.05 KB/116790 for 3d9e5bf7331338c5e5778539b0a0c908 in 632ms, sequenceid=380, compaction requested=false 2023-04-03 13:56:26,981 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:26,981 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:26,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:26,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:26,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:26,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:26,984 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:26,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:26,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,985 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:26,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280405a5[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:26,987 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:26,987 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:26,988 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:26,988 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:26,988 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:26,989 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,989 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:26,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:26,990 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:26,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:26,994 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:26,994 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:26,994 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:26,994 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:26,994 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:27,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,003 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:27,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:27,011 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,012 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:27,012 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:27,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:27,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530186399, max timestamp=1680530187039Num uniques 859; 2023-04-03 13:56:27,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530186399, max timestamp=1680530187039Num uniques 859; 2023-04-03 13:56:27,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:27,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:27,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530186399, max timestamp=1680530187039Num uniques 859; 2023-04-03 13:56:27,042 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-03 13:56:27,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:27,042 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:27,042 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:27,042 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:27,042 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:27,042 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:27,042 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:27,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/fbef6d14a01f41de85fde0a4040dfd92 2023-04-03 13:56:27,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530187077, max timestamp=1680530187093Num uniques 859; 2023-04-03 13:56:27,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:56:27,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530187077, max timestamp=1680530187093Num uniques 859; 2023-04-03 13:56:27,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:56:27,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530187077, max timestamp=1680530187093Num uniques 859; 2023-04-03 13:56:27,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:56:27,112 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:27,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:27,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:27,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:27,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:27,118 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:27,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:27,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:27,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,119 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:27,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:27,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:27,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:27,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:27,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:27,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:27,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:27,126 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,126 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:27,126 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:27,126 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:27,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:27,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:27,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:27,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530247127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680530187105, max timestamp=1680530187129Num uniques 1168; 2023-04-03 13:56:27,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=16848, suffix heap size=149440,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=368 2023-04-03 13:56:27,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680530187105, max timestamp=1680530187129Num uniques 1168; 2023-04-03 13:56:27,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=16848, suffix heap size=149440,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=368 2023-04-03 13:56:27,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680530187105, max timestamp=1680530187129Num uniques 1168; 2023-04-03 13:56:27,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=16848, suffix heap size=149440,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=368 2023-04-03 13:56:27,135 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,135 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,135 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,135 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:27,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49a4ed37[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:27,138 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:27,138 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,142 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,142 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:27,142 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:27,142 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:27,142 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,142 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:27,144 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:27,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,144 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,144 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:27,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:27,149 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:27,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:27,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:27,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:27,149 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:27,158 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,158 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,158 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,159 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,166 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,171 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530187155, max timestamp=1680530187168Num uniques 1227; 2023-04-03 13:56:27,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,171 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=12682, suffix heap size=155576,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=277 2023-04-03 13:56:27,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:27,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:27,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,172 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,172 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530187155, max timestamp=1680530187168Num uniques 1227; 2023-04-03 13:56:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=12682, suffix heap size=155576,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=277 2023-04-03 13:56:27,172 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:27,172 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:27,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530187155, max timestamp=1680530187168Num uniques 1227; 2023-04-03 13:56:27,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=12682, suffix heap size=155576,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=277 2023-04-03 13:56:27,235 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680530187184, max timestamp=1680530187213Num uniques 1136; 2023-04-03 13:56:27,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-03 13:56:27,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680530187184, max timestamp=1680530187213Num uniques 1136; 2023-04-03 13:56:27,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-04-03 13:56:27,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530187205, max timestamp=1680530187213Num uniques 1136; 2023-04-03 13:56:27,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-03 13:56:27,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680530187250, max timestamp=1680530187267Num uniques 1095; 2023-04-03 13:56:27,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680530187250, max timestamp=1680530187267Num uniques 1095; 2023-04-03 13:56:27,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-03 13:56:27,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680530187250, max timestamp=1680530187267Num uniques 1045; 2023-04-03 13:56:27,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-03 13:56:27,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=15800, suffix heap size=141848,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=345 2023-04-03 13:56:27,272 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:27,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:27,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:27,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:27,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:27,278 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:27,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:27,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:27,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,279 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:27,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:27,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:27,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:27,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:27,283 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,283 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:27,283 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:27,283 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:27,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:27,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,285 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,286 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,286 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:27,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24033cd0[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:27,290 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:27,290 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:27,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:27,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:27,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:27,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,293 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:27,293 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,293 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:27,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:27,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:27,296 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:27,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:27,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:27,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:27,296 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:27,301 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,301 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,301 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680530187287, max timestamp=1680530187300Num uniques 1204; 2023-04-03 13:56:27,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-03 13:56:27,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680530187287, max timestamp=1680530187300Num uniques 1204; 2023-04-03 13:56:27,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-03 13:56:27,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680530187287, max timestamp=1680530187300Num uniques 1204; 2023-04-03 13:56:27,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-03 13:56:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,316 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,316 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:27,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:27,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:27,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,318 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:27,318 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:27,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680530187322, max timestamp=1680530187328Num uniques 1113; 2023-04-03 13:56:27,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680530187322, max timestamp=1680530187328Num uniques 1113; 2023-04-03 13:56:27,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-03 13:56:27,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680530187322, max timestamp=1680530187328Num uniques 1113; 2023-04-03 13:56:27,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-03 13:56:27,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=12038, suffix heap size=143720,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=263 2023-04-03 13:56:27,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530247375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680530187370, max timestamp=1680530187376Num uniques 1122; 2023-04-03 13:56:27,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-03 13:56:27,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680530187370, max timestamp=1680530187376Num uniques 1122; 2023-04-03 13:56:27,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680530187370, max timestamp=1680530187376Num uniques 1122; 2023-04-03 13:56:27,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-03 13:56:27,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-04-03 13:56:27,418 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:27,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:27,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=431, cellSize=19736, totalHeapSize=45064, min timestamp=1680530187375, max timestamp=1680530187418Num uniques 1081; 2023-04-03 13:56:27,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=19736, suffix heap size=140392,new segment heap size=45064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=431 2023-04-03 13:56:27,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=431, cellSize=19736, totalHeapSize=45064, min timestamp=1680530187375, max timestamp=1680530187418Num uniques 1081; 2023-04-03 13:56:27,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=19736, suffix heap size=140392,new segment heap size=45064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=431 2023-04-03 13:56:27,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=431, cellSize=19736, totalHeapSize=45064, min timestamp=1680530187375, max timestamp=1680530187418Num uniques 1081; 2023-04-03 13:56:27,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=19736, suffix heap size=140392,new segment heap size=45064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=431 2023-04-03 13:56:27,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:27,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:27,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:27,423 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:27,423 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,423 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:27,423 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:27,424 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,424 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,425 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,425 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,425 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,425 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:27,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:27,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:27,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:27,427 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:27,428 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,428 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:27,428 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:27,428 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:27,433 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,433 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,433 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,433 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:27,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71c34b31[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:27,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:27,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,444 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:27,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:27,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:27,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:27,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530247443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:27,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:27,451 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530247445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530247447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530247447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:27,459 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:27,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:27,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:27,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:27,459 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:27,462 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,462 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,462 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,465 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,467 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:27,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:27,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:27,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,468 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:27,468 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:27,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530247483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/d173dac546594a73b85c562d6733415a 2023-04-03 13:56:27,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530247550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530247553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530247553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530247554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,568 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:27,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:27,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:27,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:27,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:27,572 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:27,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:27,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:27,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,576 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:27,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:27,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:27,577 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,578 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:27,578 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:27,578 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,578 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:27,578 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:27,578 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,578 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:27,578 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:27,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,583 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,587 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,588 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79624181[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:27,589 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:27,589 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,592 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:27,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:27,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:27,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:27,593 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:27,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:27,595 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:27,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:27,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:27,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:27,595 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:27,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,599 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:27,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:27,619 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:27,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,619 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:27,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530247691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,719 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:27,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:27,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:27,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:27,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:27,727 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:27,727 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,727 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:27,727 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:27,728 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,728 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,729 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,729 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,730 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,730 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:27,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:27,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:27,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:27,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:27,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:27,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:27,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:27,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:27,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:27,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:27,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:27,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:27,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:27,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:27,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c1c6cf[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:27,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:27,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,747 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:27,747 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:27,748 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:27,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:27,751 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:27,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:27,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:27,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:27,752 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:27,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530247755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530247757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:27,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530247759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530247759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,764 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,768 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:27,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:27,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:27,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:27,777 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,777 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,777 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,777 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,777 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,777 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:27,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:27,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:27,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:27,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:27,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:27,881 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:27,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:27,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:27,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,882 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:27,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:27,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:27,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:27,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:27,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:27,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:27,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:27,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:27,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:27,887 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:27,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a198d5b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:27,887 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:27,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:27,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:27,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:27,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:27,889 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:27,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:27,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:27,895 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:27,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:27,896 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:27,896 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:27,896 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:27,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:27,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:27,902 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,903 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,909 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:27,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:27,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:27,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:27,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:27,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:27,910 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:27,910 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:27,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:27,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=402 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/a020494a82d24e058eae59b6c714e1ac 2023-04-03 13:56:27,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/fbef6d14a01f41de85fde0a4040dfd92 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fbef6d14a01f41de85fde0a4040dfd92 2023-04-03 13:56:27,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fbef6d14a01f41de85fde0a4040dfd92, entries=200, sequenceid=402, filesize=14.3 K 2023-04-03 13:56:27,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/d173dac546594a73b85c562d6733415a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d173dac546594a73b85c562d6733415a 2023-04-03 13:56:27,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d173dac546594a73b85c562d6733415a, entries=150, sequenceid=402, filesize=11.9 K 2023-04-03 13:56:27,963 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/a020494a82d24e058eae59b6c714e1ac as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a020494a82d24e058eae59b6c714e1ac 2023-04-03 13:56:27,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a020494a82d24e058eae59b6c714e1ac, entries=150, sequenceid=402, filesize=11.9 K 2023-04-03 13:56:27,975 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=167.72 KB/171750 for 3d9e5bf7331338c5e5778539b0a0c908 in 933ms, sequenceid=402, compaction requested=true 2023-04-03 13:56:27,975 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:27,975 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:27,975 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:27,976 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:27,976 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:27,976 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:27,978 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41746 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:27,979 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/A is initiating minor compaction (all files) 2023-04-03 13:56:27,979 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/A in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,979 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:27,979 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/023e2ac38d834b7d9b6d42bc5add964d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/5d833c246320494f99e4e4a4add013e4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fbef6d14a01f41de85fde0a4040dfd92] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=40.8 K 2023-04-03 13:56:27,979 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/B is initiating minor compaction (all files) 2023-04-03 13:56:27,979 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/B in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:27,979 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c2fd3021d7a4bbbbb8ab092d72c9556, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bdd9b9748a8b4235a483f2878a1c001c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d173dac546594a73b85c562d6733415a] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=36.0 K 2023-04-03 13:56:27,980 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 023e2ac38d834b7d9b6d42bc5add964d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680530183841 2023-04-03 13:56:27,981 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2c2fd3021d7a4bbbbb8ab092d72c9556, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680530183841 2023-04-03 13:56:27,982 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5d833c246320494f99e4e4a4add013e4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530185199 2023-04-03 13:56:27,982 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bdd9b9748a8b4235a483f2878a1c001c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530185199 2023-04-03 13:56:27,982 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting fbef6d14a01f41de85fde0a4040dfd92, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680530186399 2023-04-03 13:56:27,983 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d173dac546594a73b85c562d6733415a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680530186399 2023-04-03 13:56:27,993 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#B#compaction#37 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:27,993 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#A#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:27,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680530187443, max timestamp=1680530187996Num uniques 1290; 2023-04-03 13:56:27,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=59080, new segment data size=8700, suffix heap size=162128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1290, new segment cells count=190 2023-04-03 13:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680530187443, max timestamp=1680530187996Num uniques 1290; 2023-04-03 13:56:28,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=59080, new segment data size=8700, suffix heap size=162128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1290, new segment cells count=190 2023-04-03 13:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680530187443, max timestamp=1680530187996Num uniques 1290; 2023-04-03 13:56:28,001 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=76.04 KB heapSize=202.13 KB 2023-04-03 13:56:28,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=59080, new segment data size=8700, suffix heap size=162128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1290, new segment cells count=190 2023-04-03 13:56:28,001 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:28,001 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:28,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:28,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:28,002 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:28,002 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:28,010 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:28,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:28,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:28,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:28,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:28,019 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:28,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:28,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:28,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,020 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,020 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,021 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,021 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:28,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:28,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:28,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:28,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:28,023 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,023 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:28,023 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:28,023 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:28,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:28,027 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,027 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,028 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,028 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,030 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/07ec599ab54b4a74aeb18561e97e59ed as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/07ec599ab54b4a74aeb18561e97e59ed 2023-04-03 13:56:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e8c1ea[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:28,031 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:28,032 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,036 WARN [DataStreamer for file /user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/9db906e8ae69422b95676560b55257e7] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-03 13:56:28,036 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/9db906e8ae69422b95676560b55257e7 2023-04-03 13:56:28,037 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:28,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:28,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:28,037 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,037 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:28,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:28,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:28,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,046 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/A of 3d9e5bf7331338c5e5778539b0a0c908 into 07ec599ab54b4a74aeb18561e97e59ed(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:28,046 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:28,046 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/A, priority=13, startTime=1680530187975; duration=0sec 2023-04-03 13:56:28,046 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:28,046 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:28,049 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:28,049 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:28,049 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:28,049 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:28,049 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:28,051 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:28,051 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/C is initiating minor compaction (all files) 2023-04-03 13:56:28,051 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/C in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,052 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/24177b567d1f4bce8477203eb49c3422, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/6f408cd02f394852ab6642895d718d9b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a020494a82d24e058eae59b6c714e1ac] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=36.0 K 2023-04-03 13:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,052 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,054 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:28,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,055 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 24177b567d1f4bce8477203eb49c3422, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680530183841 2023-04-03 13:56:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,055 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,055 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 6f408cd02f394852ab6642895d718d9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530185199 2023-04-03 13:56:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,056 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting a020494a82d24e058eae59b6c714e1ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680530186399 2023-04-03 13:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,058 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/2785eb0e58c0452bbbea7712b741dbde 2023-04-03 13:56:28,068 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,068 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,068 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:28,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:28,068 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:28,068 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:28,068 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,068 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:28,069 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:28,068 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:28,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:28,072 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#C#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:28,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530188064, max timestamp=1680530188070Num uniques 859; 2023-04-03 13:56:28,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:28,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530188064, max timestamp=1680530188070Num uniques 859; 2023-04-03 13:56:28,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:28,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530188064, max timestamp=1680530188070Num uniques 859; 2023-04-03 13:56:28,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:28,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530188082, max timestamp=1680530188094Num uniques 1018; 2023-04-03 13:56:28,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530188082, max timestamp=1680530188094Num uniques 1018; 2023-04-03 13:56:28,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 13:56:28,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 13:56:28,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530188082, max timestamp=1680530188094Num uniques 1018; 2023-04-03 13:56:28,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 13:56:28,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530188103, max timestamp=1680530188112Num uniques 1027; 2023-04-03 13:56:28,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530188103, max timestamp=1680530188112Num uniques 1027; 2023-04-03 13:56:28,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-03 13:56:28,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530188103, max timestamp=1680530188112Num uniques 1027; 2023-04-03 13:56:28,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-03 13:56:28,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-03 13:56:28,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=586, cellSize=26836, totalHeapSize=61184, min timestamp=1680530188119, max timestamp=1680530188131Num uniques 1136; 2023-04-03 13:56:28,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=586, cellSize=26836, totalHeapSize=61184, min timestamp=1680530188119, max timestamp=1680530188131Num uniques 1136; 2023-04-03 13:56:28,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=26836, suffix heap size=146112,new segment heap size=61184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=586 2023-04-03 13:56:28,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=26836, suffix heap size=146112,new segment heap size=61184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=586 2023-04-03 13:56:28,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=586, cellSize=26836, totalHeapSize=61184, min timestamp=1680530188119, max timestamp=1680530188131Num uniques 1136; 2023-04-03 13:56:28,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=26836, suffix heap size=146112,new segment heap size=61184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=586 2023-04-03 13:56:28,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530248146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530248147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530248147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530248148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:36236 deadline: 1680530248149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,169 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:28,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:28,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:28,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:28,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:28,172 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:28,172 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,172 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:28,172 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:28,173 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,173 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,174 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,174 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,174 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:28,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:28,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:28,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:28,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:28,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:28,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:28,175 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,176 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:28,176 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:28,176 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:28,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:28,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:28,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:28,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,186 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:28,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:28,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a6dfe1f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:28,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,190 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,190 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:28,190 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:28,190 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:28,190 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,190 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:28,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,191 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,191 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:28,191 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:28,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:28,194 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:28,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:28,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:28,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:28,194 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:28,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,197 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,197 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:28,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:28,198 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,201 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,201 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,201 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,201 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:28,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,201 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:28,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:28,202 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,202 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,202 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,202 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,202 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:28,202 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:28,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:28,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530248250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530248251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530248251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530248251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530248254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,302 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:28,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:28,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:28,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:28,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:28,307 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:28,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:28,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:28,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,308 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,308 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,308 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,308 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:28,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:28,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:28,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:28,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:28,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:28,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:28,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:28,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:28,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:28,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:28,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:28,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:28,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:28,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:28,311 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3303b223[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:28,315 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:28,315 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,316 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:28,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:28,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:28,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:28,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:28,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:28,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:28,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:28,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:28,320 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:28,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:28,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:28,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:28,320 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:28,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:28,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:28,323 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,324 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:28,327 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:28,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:28,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,327 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:28,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:28,327 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:28,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:28,422 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/491e549d36144b2b9944d6a6fb133c80 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/491e549d36144b2b9944d6a6fb133c80 2023-04-03 13:56:28,427 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:28,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:28,430 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/B of 3d9e5bf7331338c5e5778539b0a0c908 into 491e549d36144b2b9944d6a6fb133c80(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:28,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:28,430 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:28,431 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/B, priority=13, startTime=1680530187975; duration=0sec 2023-04-03 13:56:28,431 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:28,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:28,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:28,432 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:28,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:28,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:28,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,433 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:28,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:28,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:28,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:28,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:28,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:28,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:28,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:28,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:28,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:28,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:28,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:28,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:28,450 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:28,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5516824d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:28,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:28,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:28,452 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:28,452 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,452 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:28,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:28,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:28,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:28,454 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530248457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:28,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:28,458 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:28,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:28,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:28,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:28,458 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:28,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530248457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530248457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530248458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530248458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:28,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,470 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:28,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:28,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,472 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:28,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:28,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:28,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,472 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:28,472 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:28,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:28,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:28,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/d130cfc165c94fed8574c613af08675e 2023-04-03 13:56:28,491 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/ed587517cc5744108975c8a3ad1e18c0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ed587517cc5744108975c8a3ad1e18c0 2023-04-03 13:56:28,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/9db906e8ae69422b95676560b55257e7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9db906e8ae69422b95676560b55257e7 2023-04-03 13:56:28,499 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/C of 3d9e5bf7331338c5e5778539b0a0c908 into ed587517cc5744108975c8a3ad1e18c0(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:28,499 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:28,499 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/C, priority=13, startTime=1680530187976; duration=0sec 2023-04-03 13:56:28,499 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:28,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9db906e8ae69422b95676560b55257e7, entries=200, sequenceid=577, filesize=14.3 K 2023-04-03 13:56:28,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/2785eb0e58c0452bbbea7712b741dbde as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2785eb0e58c0452bbbea7712b741dbde 2023-04-03 13:56:28,511 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2785eb0e58c0452bbbea7712b741dbde, entries=150, sequenceid=577, filesize=11.9 K 2023-04-03 13:56:28,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/d130cfc165c94fed8574c613af08675e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d130cfc165c94fed8574c613af08675e 2023-04-03 13:56:28,518 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d130cfc165c94fed8574c613af08675e, entries=150, sequenceid=577, filesize=11.9 K 2023-04-03 13:56:28,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~63.28 KB/64800, currentSize=174.43 KB/178620 for 3d9e5bf7331338c5e5778539b0a0c908 in 518ms, sequenceid=577, compaction requested=false 2023-04-03 13:56:28,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:28,573 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:28,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:28,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:28,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:28,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:28,576 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:28,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:28,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:28,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,578 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:28,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:28,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:28,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:28,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:28,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:28,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:28,579 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:28,579 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:28,579 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:28,579 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:28,579 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:28,579 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:28,579 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:28,579 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=174.43 KB heapSize=464.48 KB 2023-04-03 13:56:28,580 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:28,580 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:28,580 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:28,580 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:28,580 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:28,580 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:28,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:28,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:28,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530248761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530248764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530248764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530248765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530248766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530248869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530248869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530248869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:28,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530248870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:28,991 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/f77ec6e6f8dd42728e6ca93c2027dfbd 2023-04-03 13:56:29,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530249073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530249074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530249075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530249075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530249271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530249376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530249379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530249379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530249381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,407 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/e97b3d322e0348d2b1aa6415bbe283c7 2023-04-03 13:56:29,422 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/1e3d2e5565774c33a809afadc9d43648 2023-04-03 13:56:29,429 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/f77ec6e6f8dd42728e6ca93c2027dfbd as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f77ec6e6f8dd42728e6ca93c2027dfbd 2023-04-03 13:56:29,435 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f77ec6e6f8dd42728e6ca93c2027dfbd, entries=150, sequenceid=666, filesize=11.9 K 2023-04-03 13:56:29,438 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/e97b3d322e0348d2b1aa6415bbe283c7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e97b3d322e0348d2b1aa6415bbe283c7 2023-04-03 13:56:29,444 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e97b3d322e0348d2b1aa6415bbe283c7, entries=150, sequenceid=666, filesize=11.9 K 2023-04-03 13:56:29,446 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/1e3d2e5565774c33a809afadc9d43648 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/1e3d2e5565774c33a809afadc9d43648 2023-04-03 13:56:29,455 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/1e3d2e5565774c33a809afadc9d43648, entries=150, sequenceid=666, filesize=11.9 K 2023-04-03 13:56:29,456 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~464.44 KB/475584, currentSize=26.84 KB/27480 for 3d9e5bf7331338c5e5778539b0a0c908 in 877ms, sequenceid=666, compaction requested=true 2023-04-03 13:56:29,456 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:29,456 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,458 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:29,458 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:29,459 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:29,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:29,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:29,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:29,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:29,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:29,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:29,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5276ee94[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:29,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:29,463 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:29,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:29,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:29,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:29,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:29,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:29,464 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:29,464 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:29,464 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:29,464 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:29,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:29,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:29,468 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:29,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:29,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:29,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:29,468 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:29,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:29,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:29,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:29,474 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,475 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:29,485 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:29,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:29,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:29,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:29,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:29,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:29,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:29,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:29,486 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:29,486 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,486 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:29,486 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:29,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:29,586 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:29,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:29,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:29,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:29,590 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:29,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:29,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:29,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:29,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:29,591 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:29,591 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:29,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:29,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:29,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:29,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:29,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:29,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:29,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:29,594 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:29,594 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:29,594 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:29,594 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:29,594 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 13:56:29,594 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:29,595 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:29,606 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/46191ddc9e724015a9815e8bdacf264d 2023-04-03 13:56:29,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:29,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:29,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530189928, max timestamp=1680530189932Num uniques 859; 2023-04-03 13:56:29,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530189928, max timestamp=1680530189932Num uniques 859; 2023-04-03 13:56:29,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:29,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530189928, max timestamp=1680530189932Num uniques 859; 2023-04-03 13:56:29,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:29,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:29,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680530189940, max timestamp=1680530189950Num uniques 1018; 2023-04-03 13:56:29,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 13:56:29,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680530189940, max timestamp=1680530189950Num uniques 1018; 2023-04-03 13:56:29,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 13:56:29,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680530189940, max timestamp=1680530189950Num uniques 1018; 2023-04-03 13:56:29,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 13:56:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530189963, max timestamp=1680530189969Num uniques 1127; 2023-04-03 13:56:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-04-03 13:56:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680530189957, max timestamp=1680530189969Num uniques 1127; 2023-04-03 13:56:29,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-03 13:56:29,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1680530189957, max timestamp=1680530189969Num uniques 1127; 2023-04-03 13:56:29,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-04-03 13:56:29,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530249985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530249985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:29,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530249985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:29,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680530189978, max timestamp=1680530189985Num uniques 1186; 2023-04-03 13:56:29,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-04-03 13:56:29,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680530189978, max timestamp=1680530189985Num uniques 1186; 2023-04-03 13:56:29,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=13096, suffix heap size=151312,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=286 2023-04-03 13:56:29,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680530189978, max timestamp=1680530189985Num uniques 1036; 2023-04-03 13:56:29,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-03 13:56:30,029 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/cb9589775180480d872f0ca114c15b8e 2023-04-03 13:56:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680530190001, max timestamp=1680530190031Num uniques 1145; 2023-04-03 13:56:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680530190001, max timestamp=1680530190031Num uniques 1145; 2023-04-03 13:56:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-03 13:56:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-03 13:56:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680530190001, max timestamp=1680530190031Num uniques 1145; 2023-04-03 13:56:30,034 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-04-03 13:56:30,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680530190059, max timestamp=1680530190084Num uniques 1104; 2023-04-03 13:56:30,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-03 13:56:30,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680530190059, max timestamp=1680530190084Num uniques 1104; 2023-04-03 13:56:30,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-03 13:56:30,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680530190059, max timestamp=1680530190084Num uniques 1104; 2023-04-03 13:56:30,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-03 13:56:30,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680530190098, max timestamp=1680530190107Num uniques 1063; 2023-04-03 13:56:30,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-03 13:56:30,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680530190098, max timestamp=1680530190107Num uniques 1063; 2023-04-03 13:56:30,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-03 13:56:30,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680530190098, max timestamp=1680530190107Num uniques 1063; 2023-04-03 13:56:30,109 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-03 13:56:30,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1680530190118, max timestamp=1680530190125Num uniques 1022; 2023-04-03 13:56:30,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1680530190118, max timestamp=1680530190125Num uniques 1022; 2023-04-03 13:56:30,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=19322, suffix heap size=134256,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=422 2023-04-03 13:56:30,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=422, cellSize=19322, totalHeapSize=44128, min timestamp=1680530190118, max timestamp=1680530190125Num uniques 1022; 2023-04-03 13:56:30,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=19322, suffix heap size=134256,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=422 2023-04-03 13:56:30,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=19322, suffix heap size=134256,new segment heap size=44128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=422 2023-04-03 13:56:30,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530250134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530250134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530250135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530250136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530250238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530250238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530250238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530250239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530250273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530250441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530250441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530250442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,444 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/8f06a3e671b34b2899549766e6bd91b2 2023-04-03 13:56:30,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530250442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,450 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/46191ddc9e724015a9815e8bdacf264d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/46191ddc9e724015a9815e8bdacf264d 2023-04-03 13:56:30,456 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/46191ddc9e724015a9815e8bdacf264d, entries=100, sequenceid=673, filesize=9.5 K 2023-04-03 13:56:30,457 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/cb9589775180480d872f0ca114c15b8e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/cb9589775180480d872f0ca114c15b8e 2023-04-03 13:56:30,462 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/cb9589775180480d872f0ca114c15b8e, entries=100, sequenceid=673, filesize=9.5 K 2023-04-03 13:56:30,463 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/8f06a3e671b34b2899549766e6bd91b2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/8f06a3e671b34b2899549766e6bd91b2 2023-04-03 13:56:30,470 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/8f06a3e671b34b2899549766e6bd91b2, entries=100, sequenceid=673, filesize=9.5 K 2023-04-03 13:56:30,471 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=161.02 KB/164880 for 3d9e5bf7331338c5e5778539b0a0c908 in 877ms, sequenceid=673, compaction requested=true 2023-04-03 13:56:30,471 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:30,471 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:30,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:30,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:30,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:30,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:30,473 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:30,473 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:30,474 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:30,474 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:30,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:30,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ae7fc3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:30,475 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:30,475 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:30,476 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:30,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:30,476 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:30,476 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:30,476 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:30,476 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,476 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:30,477 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:30,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,477 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:30,477 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:30,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:30,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:30,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:30,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:30,478 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:30,478 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:30,478 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:30,479 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:30,479 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:30,479 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,479 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:30,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:30,479 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:30,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:30,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:30,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:30,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:30,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:30,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:30,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,481 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:30,483 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:30,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:30,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:30,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:30,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,484 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:30,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:30,484 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:30,585 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:30,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:30,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:30,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:30,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:30,589 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:30,589 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:30,589 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:30,589 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:30,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,590 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:30,591 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,591 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:30,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:30,591 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,591 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:30,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:30,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:30,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:30,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:30,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:30,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:30,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:30,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:30,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:30,592 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:30,592 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:30,592 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=161.02 KB heapSize=440.02 KB 2023-04-03 13:56:30,593 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:30,593 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:30,593 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:30,593 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:30,593 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:30,593 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:30,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:30,748 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:30,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530250748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530250749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530250749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530250751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530250853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530250853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:30,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530250853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:30,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530250853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,008 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/9850871386274c0da2fcf1f2a60d8ac0 2023-04-03 13:56:31,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530251059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530251060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530251060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530251060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530251362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530251362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530251362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530251362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,422 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/a9c92dd2218f43e38d39037d828dd074 2023-04-03 13:56:31,844 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=829 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/569ee40e670d486cadb611c6d7f5646c 2023-04-03 13:56:31,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/9850871386274c0da2fcf1f2a60d8ac0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9850871386274c0da2fcf1f2a60d8ac0 2023-04-03 13:56:31,856 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9850871386274c0da2fcf1f2a60d8ac0, entries=150, sequenceid=829, filesize=11.9 K 2023-04-03 13:56:31,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/a9c92dd2218f43e38d39037d828dd074 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/a9c92dd2218f43e38d39037d828dd074 2023-04-03 13:56:31,863 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/a9c92dd2218f43e38d39037d828dd074, entries=150, sequenceid=829, filesize=11.9 K 2023-04-03 13:56:31,864 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/569ee40e670d486cadb611c6d7f5646c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/569ee40e670d486cadb611c6d7f5646c 2023-04-03 13:56:31,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530251864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530251866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530251867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,869 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/569ee40e670d486cadb611c6d7f5646c, entries=150, sequenceid=829, filesize=11.9 K 2023-04-03 13:56:31,870 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~439.97 KB/450528, currentSize=26.84 KB/27480 for 3d9e5bf7331338c5e5778539b0a0c908 in 1278ms, sequenceid=829, compaction requested=true 2023-04-03 13:56:31,870 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:31,870 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:31,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:31,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:31,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:31,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:31,871 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,871 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:31,872 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:31,872 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:31,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:31,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@399245bf[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:31,873 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:31,873 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:31,874 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:31,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:31,874 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:31,874 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,874 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:31,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:31,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:31,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:31,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:56:31,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:31,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:31,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:31,877 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:31,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:31,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:31,877 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:31,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:31,877 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:31,877 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:31,877 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:31,880 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:31,880 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:31,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:31,880 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:31,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:31,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,887 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:31,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:31,893 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:31,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:31,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:31,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:31,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:31,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:31,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:31,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:31,894 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:31,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:31,894 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:31,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:31,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:31,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530191902, max timestamp=1680530191921Num uniques 859; 2023-04-03 13:56:31,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530191902, max timestamp=1680530191921Num uniques 859; 2023-04-03 13:56:31,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:56:31,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:56:31,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530191902, max timestamp=1680530191921Num uniques 859; 2023-04-03 13:56:31,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:56:31,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:31,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530251967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:31,994 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:31,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:31,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:32,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:32,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:32,004 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:32,004 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,004 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:32,004 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:32,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,005 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:32,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:32,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:32,014 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:32,014 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,014 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:32,014 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:32,014 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:32,014 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,014 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:32,014 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:32,014 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,015 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:32,015 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:32,015 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,015 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:32,015 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:32,024 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,024 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:32,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58f7bc2a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:32,026 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:32,026 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,026 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,027 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:32,027 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:32,027 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:32,027 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,027 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:32,028 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,028 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,028 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,028 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:32,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:32,029 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:32,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:32,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:32,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:32,029 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:32,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,031 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,032 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:32,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,034 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,034 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:32,035 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:32,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:32,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:32,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530252068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,135 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:32,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:32,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:32,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:32,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:32,137 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:32,137 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,138 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:32,138 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:32,138 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,138 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,139 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:32,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:32,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:32,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:32,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:32,140 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,140 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:32,140 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:32,140 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:32,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:32,141 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,141 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,142 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,142 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:32,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d10ebee[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:32,143 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:32,143 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,143 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,143 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:32,146 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,146 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:32,146 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:32,146 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:32,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,147 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:32,147 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,147 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:32,152 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:32,152 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:32,152 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:32,152 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:32,152 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:32,159 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,159 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,160 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,160 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,166 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,166 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,166 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,166 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,166 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,166 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,171 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:32,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:32,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,171 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:32,171 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:32,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:32,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:32,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:32,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530252280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:32,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:32,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530252282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:32,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:32,291 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:32,291 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:32,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:32,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:32,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/1fec538409734196af14df4010025d25 2023-04-03 13:56:32,295 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,295 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,295 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,295 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:32,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:32,301 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:32,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:32,304 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:32,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:32,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:32,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:32,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:32,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:32,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:32,311 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:32,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5924dbe8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:32,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:32,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:32,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:32,314 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,314 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:32,314 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:32,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:32,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:32,316 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:32,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:32,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:32,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:32,316 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:32,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,321 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:32,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:32,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,322 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:32,322 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:32,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:32,422 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:32,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:32,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:32,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:32,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:32,426 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:32,426 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,426 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:32,426 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:32,426 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,426 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,427 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,427 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,427 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,427 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:32,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:32,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:32,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:32,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:32,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:32,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:32,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:32,428 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,428 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:32,429 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:32,429 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:32,429 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:32,429 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:32,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:32,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72b0d2ce[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:32,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:32,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,433 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,433 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:32,433 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:32,433 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:32,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:32,434 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:32,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:32,437 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:32,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:32,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:32,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:32,437 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:32,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,439 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,459 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,459 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,460 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:32,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:32,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,460 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,460 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:32,460 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:32,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:32,560 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:32,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:32,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:32,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:32,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:32,564 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:32,564 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,564 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:32,564 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:32,564 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,564 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,565 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,565 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,566 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,566 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:32,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:32,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:32,567 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:32,567 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:32,567 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:32,567 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:32,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:32,568 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,568 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,568 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c3265e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:32,569 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:32,570 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,570 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:32,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:32,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:32,570 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,570 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:32,571 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:32,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:32,573 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:32,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:32,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:32,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:32,573 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:32,573 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,574 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,575 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,576 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,576 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,576 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:32,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:32,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:32,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,577 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:32,577 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:32,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:32,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530252581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,677 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:32,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:32,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:32,680 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:32,680 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,680 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:32,680 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:32,681 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,681 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,681 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,682 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,682 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,682 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:32,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:32,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:32,683 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,683 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:32,683 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:32,683 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:32,683 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:32,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,684 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,684 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,684 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:32,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d36f0d3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:32,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:32,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,686 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,686 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:32,686 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:32,686 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:32,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:32,687 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:32,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:32,689 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:32,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:32,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:32,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:32,689 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:32,690 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,690 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,690 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,694 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:32,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:32,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,694 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:32,694 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:32,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/601a7bdb4d34413c8dfa721c6fa90f07 2023-04-03 13:56:32,794 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:32,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:32,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:32,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:32,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:32,797 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:32,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:32,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:32,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,798 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:32,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:32,799 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:32,799 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,799 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:32,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:32,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:32,800 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,800 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:32,800 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:32,800 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:32,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:32,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:32,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:32,801 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:32,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b28506[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:32,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:32,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,804 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:32,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:32,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:32,804 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,804 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:32,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,805 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:32,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:32,810 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:32,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:32,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:32,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:32,810 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:32,826 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,826 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,826 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,828 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:32,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,833 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,833 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:32,833 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:32,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:32,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530252873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:32,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530252874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:32,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530252877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,933 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:32,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:32,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:32,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:32,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:32,936 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:32,936 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:32,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:32,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,938 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,938 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,938 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,938 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:32,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:32,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:32,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:32,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:32,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:32,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:32,939 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,939 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:32,939 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:32,939 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:32,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:32,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:32,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,940 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:32,940 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:32,941 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,941 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,941 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:32,942 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:32,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26601b3c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:32,944 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:32,944 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:32,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:32,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:32,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:32,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:32,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:32,947 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:32,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:32,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:32,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:32,948 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:32,948 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,948 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:32,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,948 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,949 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,950 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:32,952 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:32,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:32,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:32,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:32,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:32,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:32,952 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:32,952 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:33,052 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:33,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:33,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:33,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:33,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:33,055 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:33,055 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:33,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:33,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,057 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,057 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,057 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,057 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:33,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:33,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:33,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:33,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:33,058 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,058 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:33,058 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:33,058 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:33,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:33,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,059 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,060 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,060 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:33,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@289f3878[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:33,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:33,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,061 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,061 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:33,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:33,061 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:33,061 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:33,062 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:33,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:33,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:33,064 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:33,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:33,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:33,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:33,064 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:33,065 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,065 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,065 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,066 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,067 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:33,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:33,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:33,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,070 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:33,070 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:33,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:33,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530253087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=839 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/3eb20929a5ed4edcbd1471811ef48b12 2023-04-03 13:56:33,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/1fec538409734196af14df4010025d25 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/1fec538409734196af14df4010025d25 2023-04-03 13:56:33,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/1fec538409734196af14df4010025d25, entries=250, sequenceid=839, filesize=16.7 K 2023-04-03 13:56:33,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/601a7bdb4d34413c8dfa721c6fa90f07 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/601a7bdb4d34413c8dfa721c6fa90f07 2023-04-03 13:56:33,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/601a7bdb4d34413c8dfa721c6fa90f07, entries=150, sequenceid=839, filesize=11.9 K 2023-04-03 13:56:33,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/3eb20929a5ed4edcbd1471811ef48b12 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/3eb20929a5ed4edcbd1471811ef48b12 2023-04-03 13:56:33,151 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/3eb20929a5ed4edcbd1471811ef48b12, entries=150, sequenceid=839, filesize=11.9 K 2023-04-03 13:56:33,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 3d9e5bf7331338c5e5778539b0a0c908 in 1275ms, sequenceid=839, compaction requested=true 2023-04-03 13:56:33,152 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,152 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:33,152 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:56:33,152 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:56:33,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:33,153 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:33,155 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71082 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:56:33,155 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/B is initiating minor compaction (all files) 2023-04-03 13:56:33,155 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/B in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,156 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/491e549d36144b2b9944d6a6fb133c80, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2785eb0e58c0452bbbea7712b741dbde, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e97b3d322e0348d2b1aa6415bbe283c7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/cb9589775180480d872f0ca114c15b8e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/a9c92dd2218f43e38d39037d828dd074, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/601a7bdb4d34413c8dfa721c6fa90f07] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=69.4 K 2023-04-03 13:56:33,156 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 491e549d36144b2b9944d6a6fb133c80, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680530186399 2023-04-03 13:56:33,156 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 78402 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:56:33,157 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/A is initiating minor compaction (all files) 2023-04-03 13:56:33,157 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/A in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,157 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/07ec599ab54b4a74aeb18561e97e59ed, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9db906e8ae69422b95676560b55257e7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f77ec6e6f8dd42728e6ca93c2027dfbd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/46191ddc9e724015a9815e8bdacf264d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9850871386274c0da2fcf1f2a60d8ac0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/1fec538409734196af14df4010025d25] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=76.6 K 2023-04-03 13:56:33,157 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 07ec599ab54b4a74aeb18561e97e59ed, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680530186399 2023-04-03 13:56:33,158 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2785eb0e58c0452bbbea7712b741dbde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680530187443 2023-04-03 13:56:33,158 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9db906e8ae69422b95676560b55257e7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680530187443 2023-04-03 13:56:33,158 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e97b3d322e0348d2b1aa6415bbe283c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680530188145 2023-04-03 13:56:33,159 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting cb9589775180480d872f0ca114c15b8e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1680530188764 2023-04-03 13:56:33,159 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f77ec6e6f8dd42728e6ca93c2027dfbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680530188145 2023-04-03 13:56:33,159 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting a9c92dd2218f43e38d39037d828dd074, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680530190133 2023-04-03 13:56:33,159 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 601a7bdb4d34413c8dfa721c6fa90f07, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680530191871 2023-04-03 13:56:33,160 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 46191ddc9e724015a9815e8bdacf264d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1680530188764 2023-04-03 13:56:33,160 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9850871386274c0da2fcf1f2a60d8ac0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680530190133 2023-04-03 13:56:33,161 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 1fec538409734196af14df4010025d25, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680530190748 2023-04-03 13:56:33,170 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:33,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:33,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:33,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:33,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:33,182 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:33,182 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,182 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:33,182 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:33,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:33,185 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#A#compaction#54 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:33,192 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#B#compaction#55 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:33,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:33,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:33,194 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:33,194 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,194 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:33,198 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:33,198 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:33,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:33,198 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-03 13:56:33,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:33,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:33,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:33,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:33,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:33,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:33,220 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/2ef65945d42849a4a4224a5bab111336 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/2ef65945d42849a4a4224a5bab111336 2023-04-03 13:56:33,220 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/40dcd617c42e497990afda6eb7c8cb14 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/40dcd617c42e497990afda6eb7c8cb14 2023-04-03 13:56:33,223 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=874 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/4bbd82ad053f4fa893444e3d56ef99bc 2023-04-03 13:56:33,230 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/A of 3d9e5bf7331338c5e5778539b0a0c908 into 2ef65945d42849a4a4224a5bab111336(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:33,230 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,230 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/A, priority=10, startTime=1680530193152; duration=0sec 2023-04-03 13:56:33,230 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:33,230 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:56:33,230 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/B of 3d9e5bf7331338c5e5778539b0a0c908 into 40dcd617c42e497990afda6eb7c8cb14(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:33,230 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,230 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/B, priority=10, startTime=1680530193152; duration=0sec 2023-04-03 13:56:33,230 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:33,232 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71082 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:56:33,232 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/C is initiating minor compaction (all files) 2023-04-03 13:56:33,232 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/C in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,233 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ed587517cc5744108975c8a3ad1e18c0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d130cfc165c94fed8574c613af08675e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/1e3d2e5565774c33a809afadc9d43648, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/8f06a3e671b34b2899549766e6bd91b2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/569ee40e670d486cadb611c6d7f5646c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/3eb20929a5ed4edcbd1471811ef48b12] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=69.4 K 2023-04-03 13:56:33,233 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting ed587517cc5744108975c8a3ad1e18c0, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=402, earliestPutTs=1680530186399 2023-04-03 13:56:33,233 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting d130cfc165c94fed8574c613af08675e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1680530187443 2023-04-03 13:56:33,234 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 1e3d2e5565774c33a809afadc9d43648, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680530188145 2023-04-03 13:56:33,234 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 8f06a3e671b34b2899549766e6bd91b2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1680530188764 2023-04-03 13:56:33,235 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 569ee40e670d486cadb611c6d7f5646c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=829, earliestPutTs=1680530190133 2023-04-03 13:56:33,235 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3eb20929a5ed4edcbd1471811ef48b12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680530191871 2023-04-03 13:56:33,257 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#C#compaction#58 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:33,292 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=874 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/351b09f1718d40b6aa00cdf380b0cfcb 2023-04-03 13:56:33,308 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=874 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/833dd31f23b6428f84dc49025e597056 2023-04-03 13:56:33,313 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/4bbd82ad053f4fa893444e3d56ef99bc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4bbd82ad053f4fa893444e3d56ef99bc 2023-04-03 13:56:33,319 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4bbd82ad053f4fa893444e3d56ef99bc, entries=150, sequenceid=874, filesize=11.9 K 2023-04-03 13:56:33,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/351b09f1718d40b6aa00cdf380b0cfcb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/351b09f1718d40b6aa00cdf380b0cfcb 2023-04-03 13:56:33,326 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/351b09f1718d40b6aa00cdf380b0cfcb, entries=150, sequenceid=874, filesize=11.9 K 2023-04-03 13:56:33,327 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/833dd31f23b6428f84dc49025e597056 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/833dd31f23b6428f84dc49025e597056 2023-04-03 13:56:33,333 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/833dd31f23b6428f84dc49025e597056, entries=150, sequenceid=874, filesize=11.9 K 2023-04-03 13:56:33,334 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=0 B/0 for 3d9e5bf7331338c5e5778539b0a0c908 in 136ms, sequenceid=874, compaction requested=false 2023-04-03 13:56:33,334 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,334 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:33,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:33,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:33,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:33,336 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:33,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3235e5d0[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:33,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:33,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,338 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:33,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:33,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:33,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:33,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:33,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:33,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:33,342 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:33,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:33,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:33,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:33,342 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:33,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,348 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:33,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,348 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:33,348 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:33,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:33,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,448 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:33,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:33,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:33,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:33,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:33,451 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:33,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:33,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:33,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,452 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,452 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,452 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:33,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:33,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:33,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:33,453 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:33,453 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,453 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:33,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:33,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:33,454 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,454 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:33,454 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,454 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:33,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:33,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:33,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:33,455 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,455 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,456 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,456 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:33,457 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:33,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74e9cc25[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:33,457 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,458 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,458 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:33,458 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:33,458 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:33,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:33,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:33,459 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:33,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:33,461 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:33,461 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:33,461 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:33,461 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:33,461 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:33,462 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,462 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,463 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:33,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:33,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:33,468 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:33,468 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:33,568 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:33,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:33,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:33,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:33,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:33,571 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:33,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:33,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:33,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,572 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:33,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:33,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:33,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:33,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:33,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:33,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:33,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:33,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:33,574 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,574 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:33,574 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,575 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:33,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:33,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:33,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:33,576 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,576 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,576 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,576 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:33,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5276807f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:33,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:33,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,579 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,579 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:33,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:33,579 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:33,579 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:33,580 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,580 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,580 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,580 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:33,582 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:33,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:33,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:33,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:33,582 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:33,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,584 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,585 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:33,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:33,587 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,587 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:33,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,587 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:33,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:33,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:33,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:33,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:33,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:33,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:33,691 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:33,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:33,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:33,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,695 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,696 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,696 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,696 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:33,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:33,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:33,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:33,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:33,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:33,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:33,697 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,698 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:33,698 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,698 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:33,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:33,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:33,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:33,698 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/7088fdeb21014ee7b927a6c5d53afa08 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7088fdeb21014ee7b927a6c5d53afa08 2023-04-03 13:56:33,699 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,699 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,699 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,699 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:33,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:33,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fb2eb1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:33,701 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:33,701 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,702 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,702 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:33,702 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:33,702 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:33,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:33,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:33,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:33,708 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:33,708 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:33,708 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:33,708 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:33,708 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:33,711 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/C of 3d9e5bf7331338c5e5778539b0a0c908 into 7088fdeb21014ee7b927a6c5d53afa08(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:33,711 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,711 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/C, priority=10, startTime=1680530193153; duration=0sec 2023-04-03 13:56:33,711 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:33,717 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,717 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,717 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,718 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,721 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,721 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,721 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:33,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:33,721 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,722 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:33,722 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:33,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:33,721 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,721 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,722 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,722 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,722 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,722 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,822 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:33,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:33,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:33,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:33,825 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:33,825 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,825 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:33,825 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:33,825 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,825 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,826 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,826 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,826 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,826 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:33,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:33,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:33,827 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:33,827 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,827 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:33,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:33,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:33,828 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,828 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:33,828 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,828 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:33,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:33,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:33,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:33,829 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e7d407b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:33,831 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:33,831 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,832 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,832 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:33,832 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:33,832 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:33,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:33,832 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:33,835 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:33,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:33,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:33,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:33,835 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:33,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,838 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,839 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:33,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,841 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,841 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,841 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,941 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:33,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:33,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:33,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:33,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:33,945 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:33,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:33,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:33,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,946 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,946 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,946 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:33,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:33,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:33,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:33,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:33,947 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:33,947 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:33,947 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:33,947 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:33,947 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:33,947 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,947 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:33,947 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:33,947 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:33,947 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:33,948 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:33,948 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,948 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:33,948 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:33,948 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,948 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:33,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24cf2f75[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:33,950 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:33,950 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,951 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:33,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:33,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:33,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,952 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:33,954 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:33,954 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:33,954 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:33,954 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:33,954 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:33,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,956 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,957 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:33,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:33,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:33,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:33,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:33,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:33,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:33,959 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:33,959 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:33,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:33,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,960 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:33,960 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:33,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,060 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:34,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:34,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:34,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:34,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:34,062 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:34,062 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:34,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:34,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,064 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:34,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:34,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:34,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:34,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:34,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:34,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:34,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:34,065 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,066 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:34,066 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:34,066 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:34,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:34,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:34,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:34,067 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,067 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,067 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,067 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:34,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@425c26eb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:34,068 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:34,068 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,069 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:34,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:34,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:34,070 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,070 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:34,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:34,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:34,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:34,074 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:34,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:34,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:34,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:34,074 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:34,075 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,075 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,075 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,076 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,077 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:34,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,079 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:34,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,080 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:34,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:34,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:34,096 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:56:34,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:34,096 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:34,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:34,096 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:34,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:34,096 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:34,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530194122, max timestamp=1680530194136Num uniques 859; 2023-04-03 13:56:34,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:34,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530194122, max timestamp=1680530194136Num uniques 859; 2023-04-03 13:56:34,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:34,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530194122, max timestamp=1680530194136Num uniques 859; 2023-04-03 13:56:34,137 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:34,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:34,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530254163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,180 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:34,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:34,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:34,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:34,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:34,182 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:34,182 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:34,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:34,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:34,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:34,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:34,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:34,185 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,185 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:34,185 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:34,185 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:34,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:34,186 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,187 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,187 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188816b7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:34,188 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:34,188 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,188 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:34,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:34,189 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:34,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:34,191 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:34,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:34,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:34,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:34,192 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:34,193 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,193 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,193 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,193 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,196 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:34,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:34,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:34,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,197 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:34,197 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:34,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:34,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530254265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,297 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:34,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:34,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:34,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:34,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:34,301 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:34,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:34,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:34,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,305 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:34,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:34,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:34,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:34,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:34,309 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:34,309 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,309 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:34,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:34,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:34,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:34,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:34,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:34,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:34,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:34,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:34,311 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:34,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:34,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c02e345[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:34,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:34,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:34,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:34,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:34,314 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:34,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:34,316 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:34,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:34,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:34,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:34,317 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:34,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,328 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,328 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,328 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,328 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:34,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:34,329 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:34,329 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:34,328 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,328 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,429 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:34,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:34,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:34,432 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:34,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:34,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:34,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,433 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:34,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:34,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:34,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:34,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:34,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:34,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:34,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:34,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:34,434 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,434 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:34,435 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:34,435 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:34,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:34,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:34,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:34,436 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:34,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31ebdd37[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:34,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:34,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:34,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:34,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:34,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:34,439 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,439 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,439 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,439 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:34,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:34,441 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:34,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:34,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:34,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:34,441 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:34,442 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,442 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,444 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,446 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:34,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,447 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:34,447 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:34,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:34,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:34,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530254471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,513 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=893 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/e78a2a4b2fb5422fbea62e629e553db6 2023-04-03 13:56:34,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=893 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/e50d94b6efb64c60902812ddd8cb0e42 2023-04-03 13:56:34,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=893 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/ad85a877a56e4dcea389c29c67318f96 2023-04-03 13:56:34,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/e78a2a4b2fb5422fbea62e629e553db6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/e78a2a4b2fb5422fbea62e629e553db6 2023-04-03 13:56:34,547 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:34,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:34,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/e78a2a4b2fb5422fbea62e629e553db6, entries=200, sequenceid=893, filesize=14.3 K 2023-04-03 13:56:34,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:34,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:34,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/e50d94b6efb64c60902812ddd8cb0e42 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e50d94b6efb64c60902812ddd8cb0e42 2023-04-03 13:56:34,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:34,550 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:34,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:34,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:34,551 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,551 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,553 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,553 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,553 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:34,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:34,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:34,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:34,555 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,555 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:34,555 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:34,555 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:34,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:34,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,556 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e50d94b6efb64c60902812ddd8cb0e42, entries=150, sequenceid=893, filesize=11.9 K 2023-04-03 13:56:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,557 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,557 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/ad85a877a56e4dcea389c29c67318f96 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ad85a877a56e4dcea389c29c67318f96 2023-04-03 13:56:34,557 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,557 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:34,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12a3ce9f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:34,559 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:34,560 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,560 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:34,560 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,560 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:34,560 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:34,561 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:34,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,561 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,562 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ad85a877a56e4dcea389c29c67318f96, entries=150, sequenceid=893, filesize=11.9 K 2023-04-03 13:56:34,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:34,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:34,564 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:34,564 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:34,564 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:34,564 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:34,564 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:34,564 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3d9e5bf7331338c5e5778539b0a0c908 in 468ms, sequenceid=893, compaction requested=true 2023-04-03 13:56:34,564 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:34,564 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:34,564 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:34,564 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:34,564 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:34,565 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:34,565 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:34,565 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:34,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,565 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:34,566 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,566 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/A is initiating minor compaction (all files) 2023-04-03 13:56:34,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,566 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,566 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:34,566 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/B is initiating minor compaction (all files) 2023-04-03 13:56:34,566 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/B in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:34,567 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/40dcd617c42e497990afda6eb7c8cb14, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/351b09f1718d40b6aa00cdf380b0cfcb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e50d94b6efb64c60902812ddd8cb0e42] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=36.3 K 2023-04-03 13:56:34,566 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/A in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,580 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/2ef65945d42849a4a4224a5bab111336, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4bbd82ad053f4fa893444e3d56ef99bc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/e78a2a4b2fb5422fbea62e629e553db6] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=38.7 K 2023-04-03 13:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,580 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2ef65945d42849a4a4224a5bab111336, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680530191871 2023-04-03 13:56:34,580 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 40dcd617c42e497990afda6eb7c8cb14, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680530191871 2023-04-03 13:56:34,580 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 4bbd82ad053f4fa893444e3d56ef99bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=874, earliestPutTs=1680530191957 2023-04-03 13:56:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,580 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 351b09f1718d40b6aa00cdf380b0cfcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=874, earliestPutTs=1680530191957 2023-04-03 13:56:34,581 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting e78a2a4b2fb5422fbea62e629e553db6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=893, earliestPutTs=1680530194092 2023-04-03 13:56:34,581 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e50d94b6efb64c60902812ddd8cb0e42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=893, earliestPutTs=1680530194092 2023-04-03 13:56:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:34,581 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:34,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:34,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,585 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:34,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:34,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:34,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:34,588 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#B#compaction#64 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:34,588 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#A#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:34,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:34,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:34,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:34,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:34,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:34,688 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:34,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:34,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:34,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,689 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:34,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:34,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:34,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:34,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:34,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:34,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:34,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:34,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:34,691 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:34,691 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:34,691 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:34,691 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 13:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:34,691 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:34,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:34,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:34,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:34,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530254786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:34,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530254877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,881 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:34,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:34,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530254888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:34,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530254889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,890 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:34,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:34,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530254893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:34,897 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:35,005 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/b6895a9d38f547a58d18ede1b2dbcfe7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b6895a9d38f547a58d18ede1b2dbcfe7 2023-04-03 13:56:35,010 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/8c0b3ef947c9480ab98d32d95eace0ba as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/8c0b3ef947c9480ab98d32d95eace0ba 2023-04-03 13:56:35,012 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/A of 3d9e5bf7331338c5e5778539b0a0c908 into b6895a9d38f547a58d18ede1b2dbcfe7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:35,012 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:35,012 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/A, priority=13, startTime=1680530194564; duration=0sec 2023-04-03 13:56:35,012 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:35,012 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:35,013 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:35,013 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/C is initiating minor compaction (all files) 2023-04-03 13:56:35,013 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/C in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:35,013 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7088fdeb21014ee7b927a6c5d53afa08, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/833dd31f23b6428f84dc49025e597056, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ad85a877a56e4dcea389c29c67318f96] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=36.3 K 2023-04-03 13:56:35,013 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 7088fdeb21014ee7b927a6c5d53afa08, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=839, earliestPutTs=1680530191871 2023-04-03 13:56:35,014 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 833dd31f23b6428f84dc49025e597056, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=874, earliestPutTs=1680530191957 2023-04-03 13:56:35,014 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting ad85a877a56e4dcea389c29c67318f96, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=893, earliestPutTs=1680530194092 2023-04-03 13:56:35,018 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/B of 3d9e5bf7331338c5e5778539b0a0c908 into 8c0b3ef947c9480ab98d32d95eace0ba(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:35,018 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:35,018 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/B, priority=13, startTime=1680530194564; duration=0sec 2023-04-03 13:56:35,019 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:35,035 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#C#compaction#66 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:35,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:35,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530255091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,099 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/4ddaa006f9bf45c8866863f37c04d12b 2023-04-03 13:56:35,118 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/ad96a5898893478da09f3f082a72c74e 2023-04-03 13:56:35,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:35,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530255397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,450 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/e824d66381844187963f6280579df336 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e824d66381844187963f6280579df336 2023-04-03 13:56:35,457 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/C of 3d9e5bf7331338c5e5778539b0a0c908 into e824d66381844187963f6280579df336(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:35,457 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:35,457 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/C, priority=13, startTime=1680530194565; duration=0sec 2023-04-03 13:56:35,457 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:35,546 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/36eb8b5cdd184faba29fc9083867924d 2023-04-03 13:56:35,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/4ddaa006f9bf45c8866863f37c04d12b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4ddaa006f9bf45c8866863f37c04d12b 2023-04-03 13:56:35,556 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4ddaa006f9bf45c8866863f37c04d12b, entries=150, sequenceid=929, filesize=11.9 K 2023-04-03 13:56:35,557 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/ad96a5898893478da09f3f082a72c74e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ad96a5898893478da09f3f082a72c74e 2023-04-03 13:56:35,562 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ad96a5898893478da09f3f082a72c74e, entries=150, sequenceid=929, filesize=11.9 K 2023-04-03 13:56:35,563 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/36eb8b5cdd184faba29fc9083867924d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/36eb8b5cdd184faba29fc9083867924d 2023-04-03 13:56:35,568 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/36eb8b5cdd184faba29fc9083867924d, entries=150, sequenceid=929, filesize=11.9 K 2023-04-03 13:56:35,569 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 3d9e5bf7331338c5e5778539b0a0c908 in 878ms, sequenceid=929, compaction requested=false 2023-04-03 13:56:35,569 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:35,569 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:35,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:35,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:35,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:35,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:35,570 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:35,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:35,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:35,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:35,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:35,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:35,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:35,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:35,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:35,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:35,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a465fd2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:35,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:35,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:35,573 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:35,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:35,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:35,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:35,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:35,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:35,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:35,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:35,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:35,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:35,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:35,576 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:35,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:35,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:35,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:35,576 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:35,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:35,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:35,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:35,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:35,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:35,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:35,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:35,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:35,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:35,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,580 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,580 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:35,580 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:35,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:35,580 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:35,580 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:35,580 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:35,581 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:35,581 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:35,581 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:35,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:35,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:35,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:35,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:35,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:35,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:35,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:35,682 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:35,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:35,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:35,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:35,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:35,685 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:35,685 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:35,685 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:35,685 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:35,685 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,685 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:35,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,686 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:35,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:35,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:35,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:35,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:35,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:35,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:35,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:35,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:35,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:35,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:35,688 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:35,688 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:35,688 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:35,688 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:35,688 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:56:35,688 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:35,688 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:35,688 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:35,688 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:35,688 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:35,688 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:35,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:35,903 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:35,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530195923, max timestamp=1680530195946Num uniques 859; 2023-04-03 13:56:35,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:35,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530195923, max timestamp=1680530195946Num uniques 859; 2023-04-03 13:56:35,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:35,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530195923, max timestamp=1680530195946Num uniques 859; 2023-04-03 13:56:35,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:35,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:35,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530255975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:36,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530256077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,101 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=942 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/bd5543f2dae542eda68e02787e065db1 2023-04-03 13:56:36,113 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=942 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/68c5ca72d8a94bc698ed2f4460959154 2023-04-03 13:56:36,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:36,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530256280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:36,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36270 deadline: 1680530256307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,310 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:36,531 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=942 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/4dbc2d4b24c74bfdbefe67d0b1e59cd4 2023-04-03 13:56:36,537 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/bd5543f2dae542eda68e02787e065db1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/bd5543f2dae542eda68e02787e065db1 2023-04-03 13:56:36,541 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/bd5543f2dae542eda68e02787e065db1, entries=150, sequenceid=942, filesize=11.9 K 2023-04-03 13:56:36,542 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/68c5ca72d8a94bc698ed2f4460959154 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/68c5ca72d8a94bc698ed2f4460959154 2023-04-03 13:56:36,547 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/68c5ca72d8a94bc698ed2f4460959154, entries=150, sequenceid=942, filesize=11.9 K 2023-04-03 13:56:36,548 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/4dbc2d4b24c74bfdbefe67d0b1e59cd4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/4dbc2d4b24c74bfdbefe67d0b1e59cd4 2023-04-03 13:56:36,553 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/4dbc2d4b24c74bfdbefe67d0b1e59cd4, entries=150, sequenceid=942, filesize=11.9 K 2023-04-03 13:56:36,553 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3d9e5bf7331338c5e5778539b0a0c908 in 865ms, sequenceid=942, compaction requested=true 2023-04-03 13:56:36,553 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:36,554 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:36,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:36,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31259514[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:36,558 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:36,558 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,559 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,559 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:36,559 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:36,559 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:36,559 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,559 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:36,560 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,560 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,560 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:36,560 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:36,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:36,562 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:36,562 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:36,562 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:36,562 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:36,562 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:36,563 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,570 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,570 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:36,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:36,570 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:36,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:36,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:36,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:36,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,577 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:36,577 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:36,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:36,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:36,584 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-03 13:56:36,584 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:36,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:36,584 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:36,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:36,584 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:36,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:36,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/745328121e9b4c418bd8435d81908a60 2023-04-03 13:56:36,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:36,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530256604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,677 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:36,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:36,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:36,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:36,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:36,691 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:36,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:36,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:36,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:36,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:36,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:36,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:36,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:36,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:36,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:36,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:36,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:36,703 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:36,703 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:36,703 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:36,703 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:36,703 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:36,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,709 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,710 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,710 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530256710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e48e462[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:36,711 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:36,711 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,716 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,716 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:36,716 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:36,716 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:36,717 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,717 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:36,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:36,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:36,717 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:36,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:36,719 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:36,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:36,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:36,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:36,719 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:36,720 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,720 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,720 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:36,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,721 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,722 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:36,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:36,723 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,723 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:36,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:36,723 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:36,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:36,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:36,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:36,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:36,823 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:36,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:36,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:36,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:36,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:36,826 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:36,826 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:36,826 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:36,826 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:36,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,827 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:36,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:36,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:36,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:36,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:36,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:36,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:36,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:36,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:36,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:36,830 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:36,830 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:36,830 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:36,831 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:36,831 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:36,831 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:36,831 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,831 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:36,831 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:36,832 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,832 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,832 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,832 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:36,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52931e02[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:36,834 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:36,834 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,835 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:36,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:36,835 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,836 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:36,836 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:36,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:36,838 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:36,838 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:36,838 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:36,838 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:36,838 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:36,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:36,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:36,840 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,841 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,849 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,849 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,849 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,849 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:36,850 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:36,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:36,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:36,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:36,850 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:36,850 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:36,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:36,850 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:36,850 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,850 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:36,850 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:36,850 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:36,913 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:36,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530256912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,950 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:36,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:36,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:36,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:36,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:36,953 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:36,953 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:36,953 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:36,953 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:36,953 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,953 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,954 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,954 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:36,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:36,954 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,954 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:36,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:36,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:36,955 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:36,955 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:36,955 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:36,955 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:36,956 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:36,956 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:36,956 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:36,956 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:36,956 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:36,956 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:36,956 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:36,956 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,956 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:36,956 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:36,957 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,957 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,957 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,957 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:36,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ad3734d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:36,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:36,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,959 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:36,959 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:36,959 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:36,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:36,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,959 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,959 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:36,960 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:36,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:36,961 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:36,961 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:36,961 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:36,961 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:36,961 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:36,962 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,962 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:36,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,962 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:36,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:36,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,963 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:36,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,964 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:36,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,965 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:36,965 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:36,965 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:36,965 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:36,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:36,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:36,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:36,965 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:36,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:36,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:36,966 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:36,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:36,966 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:36,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:36,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:37,012 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/c35be200f9934bc587b2bf2bd1bd0c00 2023-04-03 13:56:37,066 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:37,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:37,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:37,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:37,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:37,069 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:37,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:37,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:37,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:37,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,071 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,071 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:37,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:37,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:37,071 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,071 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:37,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:37,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:37,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:37,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:37,072 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,072 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:37,072 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:37,072 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:37,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:37,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,073 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,073 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,073 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:37,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:37,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c3ce36[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:37,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:37,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,075 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,075 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:37,075 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:37,075 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:37,075 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,075 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:37,076 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,076 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,076 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:37,076 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:37,078 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:37,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:37,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:37,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:37,078 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:37,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,080 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,080 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:37,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:37,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,081 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,082 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:37,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:37,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:37,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:37,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:37,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:37,083 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:37,083 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,083 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:37,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:37,083 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,083 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,083 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,083 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:37,183 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:37,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:37,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:37,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:37,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:37,185 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:37,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:37,186 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:37,186 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:37,186 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,186 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,187 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:37,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:37,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:37,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,188 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:37,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:37,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:37,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:37,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:37,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:37,192 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:37,192 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:37,192 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,192 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:37,192 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:37,192 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,193 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:37,193 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:37,193 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,193 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:37,193 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:37,199 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,200 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,200 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e314758[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:37,218 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:37,218 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,219 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,219 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:37,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:37,219 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:37,219 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:37,220 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:37,220 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:37,220 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,220 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530257218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:37,222 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:37,222 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:37,222 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:37,222 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:37,222 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:37,223 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,223 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,223 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:37,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,223 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:37,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:37,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:37,226 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:37,227 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:37,227 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:37,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:37,327 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:37,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:37,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:37,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:37,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:37,330 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:37,330 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:37,330 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:37,330 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:37,330 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,330 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:37,331 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:37,331 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,331 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:37,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:37,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:37,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:37,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:37,332 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:37,332 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:37,332 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:37,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:37,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:37,333 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,333 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:37,333 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:37,333 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:37,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:37,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:37,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:37,334 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:37,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64182f92[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:37,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:37,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:37,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:37,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:37,339 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,339 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:37,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:37,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:37,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:37,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:37,342 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:37,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:37,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:37,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:37,342 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:37,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:37,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:37,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:37,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:37,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:37,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,345 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:37,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:37,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:37,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:37,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,347 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:37,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,347 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:37,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:37,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=979 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/a1e6b874500046bd86f7a631910629c4 2023-04-03 13:56:37,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/745328121e9b4c418bd8435d81908a60 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/745328121e9b4c418bd8435d81908a60 2023-04-03 13:56:37,433 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/745328121e9b4c418bd8435d81908a60, entries=200, sequenceid=979, filesize=14.3 K 2023-04-03 13:56:37,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/c35be200f9934bc587b2bf2bd1bd0c00 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/c35be200f9934bc587b2bf2bd1bd0c00 2023-04-03 13:56:37,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/c35be200f9934bc587b2bf2bd1bd0c00, entries=150, sequenceid=979, filesize=11.9 K 2023-04-03 13:56:37,439 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/a1e6b874500046bd86f7a631910629c4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a1e6b874500046bd86f7a631910629c4 2023-04-03 13:56:37,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a1e6b874500046bd86f7a631910629c4, entries=150, sequenceid=979, filesize=11.9 K 2023-04-03 13:56:37,444 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 3d9e5bf7331338c5e5778539b0a0c908 in 860ms, sequenceid=979, compaction requested=true 2023-04-03 13:56:37,444 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:37,445 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:37,445 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:37,445 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:56:37,445 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:56:37,445 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:37,446 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:56:37,446 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:56:37,446 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/B is initiating minor compaction (all files) 2023-04-03 13:56:37,446 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/A is initiating minor compaction (all files) 2023-04-03 13:56:37,446 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/B in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,446 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/A in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,446 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/8c0b3ef947c9480ab98d32d95eace0ba, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ad96a5898893478da09f3f082a72c74e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/68c5ca72d8a94bc698ed2f4460959154, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/c35be200f9934bc587b2bf2bd1bd0c00] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=48.3 K 2023-04-03 13:56:37,446 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b6895a9d38f547a58d18ede1b2dbcfe7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4ddaa006f9bf45c8866863f37c04d12b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/bd5543f2dae542eda68e02787e065db1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/745328121e9b4c418bd8435d81908a60] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=50.7 K 2023-04-03 13:56:37,447 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting b6895a9d38f547a58d18ede1b2dbcfe7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=893, earliestPutTs=1680530194092 2023-04-03 13:56:37,447 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 8c0b3ef947c9480ab98d32d95eace0ba, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=893, earliestPutTs=1680530194092 2023-04-03 13:56:37,447 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:37,447 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ad96a5898893478da09f3f082a72c74e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1680530194157 2023-04-03 13:56:37,447 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 4ddaa006f9bf45c8866863f37c04d12b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1680530194157 2023-04-03 13:56:37,448 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 68c5ca72d8a94bc698ed2f4460959154, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=942, earliestPutTs=1680530194782 2023-04-03 13:56:37,448 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting bd5543f2dae542eda68e02787e065db1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=942, earliestPutTs=1680530194782 2023-04-03 13:56:37,448 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c35be200f9934bc587b2bf2bd1bd0c00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680530195970 2023-04-03 13:56:37,448 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 745328121e9b4c418bd8435d81908a60, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680530195970 2023-04-03 13:56:37,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:37,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:37,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:37,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:37,456 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:37,456 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:37,456 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:37,456 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:37,457 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,457 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,457 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#B#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:37,457 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,457 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:37,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:37,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:37,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:37,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:37,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:37,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:37,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:37,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:37,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:37,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:37,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:37,459 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,459 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:37,460 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 13:56:37,460 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:37,460 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:37,460 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:37,460 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:37,460 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:37,460 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:37,462 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#A#compaction#76 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:37,478 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/fc6faa16eff5470f8c6c9154f0c251c0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/fc6faa16eff5470f8c6c9154f0c251c0 2023-04-03 13:56:37,482 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/b2dcd0e86d844b4f81a6b848fa4f3844 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b2dcd0e86d844b4f81a6b848fa4f3844 2023-04-03 13:56:37,485 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/B of 3d9e5bf7331338c5e5778539b0a0c908 into fc6faa16eff5470f8c6c9154f0c251c0(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:37,485 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:37,485 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/B, priority=12, startTime=1680530197445; duration=0sec 2023-04-03 13:56:37,485 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:37,485 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:56:37,488 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:56:37,488 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/C is initiating minor compaction (all files) 2023-04-03 13:56:37,488 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/C in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:37,488 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e824d66381844187963f6280579df336, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/36eb8b5cdd184faba29fc9083867924d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/4dbc2d4b24c74bfdbefe67d0b1e59cd4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a1e6b874500046bd86f7a631910629c4] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=48.3 K 2023-04-03 13:56:37,488 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e824d66381844187963f6280579df336, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=893, earliestPutTs=1680530194092 2023-04-03 13:56:37,489 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 36eb8b5cdd184faba29fc9083867924d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=929, earliestPutTs=1680530194157 2023-04-03 13:56:37,489 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4dbc2d4b24c74bfdbefe67d0b1e59cd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=942, earliestPutTs=1680530194782 2023-04-03 13:56:37,490 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting a1e6b874500046bd86f7a631910629c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680530195970 2023-04-03 13:56:37,491 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/A of 3d9e5bf7331338c5e5778539b0a0c908 into b2dcd0e86d844b4f81a6b848fa4f3844(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:37,491 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:37,491 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/A, priority=12, startTime=1680530197445; duration=0sec 2023-04-03 13:56:37,491 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:37,498 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#C#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:37,513 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/71c5fc93d1124feb9157bd1ef8cbcfde as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/71c5fc93d1124feb9157bd1ef8cbcfde 2023-04-03 13:56:37,518 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/C of 3d9e5bf7331338c5e5778539b0a0c908 into 71c5fc93d1124feb9157bd1ef8cbcfde(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:37,518 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:37,518 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/C, priority=12, startTime=1680530197445; duration=0sec 2023-04-03 13:56:37,518 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:37,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:37,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:37,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530197746, max timestamp=1680530197753Num uniques 859; 2023-04-03 13:56:37,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530197746, max timestamp=1680530197753Num uniques 859; 2023-04-03 13:56:37,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:37,754 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:37,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530197746, max timestamp=1680530197753Num uniques 859; 2023-04-03 13:56:37,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:56:37,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:37,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530257787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:37,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530257888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:37,893 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=988 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/44cbe38863a84eb3a68a7c6ac3bbe606 2023-04-03 13:56:38,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:38,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530258091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,305 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=988 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/bae8a614b9f3433a8792115bf2c9908c 2023-04-03 13:56:38,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:38,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530258394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,716 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=988 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/7888e498a6544920959b99ca7a96abe3 2023-04-03 13:56:38,721 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/44cbe38863a84eb3a68a7c6ac3bbe606 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/44cbe38863a84eb3a68a7c6ac3bbe606 2023-04-03 13:56:38,727 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/44cbe38863a84eb3a68a7c6ac3bbe606, entries=150, sequenceid=988, filesize=11.9 K 2023-04-03 13:56:38,728 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/bae8a614b9f3433a8792115bf2c9908c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bae8a614b9f3433a8792115bf2c9908c 2023-04-03 13:56:38,733 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bae8a614b9f3433a8792115bf2c9908c, entries=150, sequenceid=988, filesize=11.9 K 2023-04-03 13:56:38,734 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/7888e498a6544920959b99ca7a96abe3 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7888e498a6544920959b99ca7a96abe3 2023-04-03 13:56:38,738 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7888e498a6544920959b99ca7a96abe3, entries=150, sequenceid=988, filesize=11.9 K 2023-04-03 13:56:38,739 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 3d9e5bf7331338c5e5778539b0a0c908 in 1279ms, sequenceid=988, compaction requested=false 2023-04-03 13:56:38,739 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:38,739 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:38,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:38,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:38,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:38,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:38,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:38,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:38,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55e07147[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:38,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:38,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:38,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:38,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:38,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:38,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:38,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:38,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:38,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:38,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:38,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:38,746 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:38,746 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:38,746 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:38,746 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:38,746 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:38,747 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,747 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:38,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,747 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:38,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:38,748 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,749 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:38,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,750 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,750 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:38,750 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:38,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:38,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:38,750 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:38,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:38,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:38,750 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:38,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:38,751 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:38,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,751 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:38,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:38,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:38,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:38,851 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:38,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:38,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:38,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:38,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:38,854 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:38,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:38,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:38,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:38,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:38,855 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:38,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:38,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:38,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:38,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:38,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:38,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:38,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:38,856 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:38,856 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. started... 2023-04-03 13:56:38,856 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-03 13:56:38,856 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:38,856 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:38,856 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:38,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:38,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:38,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:38,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:38,899 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. as already flushing 2023-04-03 13:56:38,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:38,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530258904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:38,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530258905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:38,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36280 deadline: 1680530258909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,911 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:38,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:38,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36234 deadline: 1680530258915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:38,917 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:56:39,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530259006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530259006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:39,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530259208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:39,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530259208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,264 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/47fdfab908f14a2b800f564f74927420 2023-04-03 13:56:39,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:39,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36236 deadline: 1680530259510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:39,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:36252 deadline: 1680530259512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,649 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1439cba9 to 127.0.0.1:57081 2023-04-03 13:56:39,649 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:39,650 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ecabb06 to 127.0.0.1:57081 2023-04-03 13:56:39,650 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:39,654 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x655398b8 to 127.0.0.1:57081 2023-04-03 13:56:39,654 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:39,656 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a7ef79d to 127.0.0.1:57081 2023-04-03 13:56:39,656 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:39,677 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/9b1f9a9b457d462fbccc68ab3a6fa697 2023-04-03 13:56:39,692 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1028 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/b4c8605c1cda4c43ae602526060f4b39 2023-04-03 13:56:39,700 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/47fdfab908f14a2b800f564f74927420 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/47fdfab908f14a2b800f564f74927420 2023-04-03 13:56:39,708 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/47fdfab908f14a2b800f564f74927420, entries=150, sequenceid=1028, filesize=11.9 K 2023-04-03 13:56:39,709 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/9b1f9a9b457d462fbccc68ab3a6fa697 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/9b1f9a9b457d462fbccc68ab3a6fa697 2023-04-03 13:56:39,719 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/9b1f9a9b457d462fbccc68ab3a6fa697, entries=150, sequenceid=1028, filesize=11.9 K 2023-04-03 13:56:39,720 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/b4c8605c1cda4c43ae602526060f4b39 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/b4c8605c1cda4c43ae602526060f4b39 2023-04-03 13:56:39,726 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/b4c8605c1cda4c43ae602526060f4b39, entries=150, sequenceid=1028, filesize=11.9 K 2023-04-03 13:56:39,727 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 3d9e5bf7331338c5e5778539b0a0c908 in 871ms, sequenceid=1028, compaction requested=true 2023-04-03 13:56:39,727 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:39,727 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:39,727 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:39,727 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:39,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:39,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:39,729 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,729 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:39,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:39,729 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:39,730 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:39,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:39,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:39,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cfba7b3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:39,732 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:39,732 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:39,733 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:39,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:39,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:39,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:39,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:39,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:39,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:39,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:39,734 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:39,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:39,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:39,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:39,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:39,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:39,737 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:39,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:39,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:39,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:39,737 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:39,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:39,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:39,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:39,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:39,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:39,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:39,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:39,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:39,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:39,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:39,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:39,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:39,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:39,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:39,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:39,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:39,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:39,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:39,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:39,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:39,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:39,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:39,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:39,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:39,742 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:39,742 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:39,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:39,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:39,842 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:39,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:40,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:40,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:56:40,014 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f537ba4 to 127.0.0.1:57081 2023-04-03 13:56:40,014 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:40,014 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:40,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:40,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:40,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:40,015 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:40,015 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:40,018 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33e5a282 to 127.0.0.1:57081 2023-04-03 13:56:40,018 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:40,424 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1038 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/ff612675c25a44278385f7e1c652c428 2023-04-03 13:56:40,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1038 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/5b22fb7cc749400abc2e807a8a88fbb4 2023-04-03 13:56:41,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1038 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/11fbf4f0d13f46318f68c92db588bb36 2023-04-03 13:56:41,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/ff612675c25a44278385f7e1c652c428 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/ff612675c25a44278385f7e1c652c428 2023-04-03 13:56:41,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/ff612675c25a44278385f7e1c652c428, entries=150, sequenceid=1038, filesize=11.9 K 2023-04-03 13:56:41,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/5b22fb7cc749400abc2e807a8a88fbb4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/5b22fb7cc749400abc2e807a8a88fbb4 2023-04-03 13:56:41,290 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/5b22fb7cc749400abc2e807a8a88fbb4, entries=150, sequenceid=1038, filesize=11.9 K 2023-04-03 13:56:41,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/11fbf4f0d13f46318f68c92db588bb36 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/11fbf4f0d13f46318f68c92db588bb36 2023-04-03 13:56:41,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/11fbf4f0d13f46318f68c92db588bb36, entries=150, sequenceid=1038, filesize=11.9 K 2023-04-03 13:56:41,296 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=6.71 KB/6870 for 3d9e5bf7331338c5e5778539b0a0c908 in 1282ms, sequenceid=1038, compaction requested=true 2023-04-03 13:56:41,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:41,296 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:56:41,296 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:41,296 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:56:41,297 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:56:41,297 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/B is initiating minor compaction (all files) 2023-04-03 13:56:41,297 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:56:41,297 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/B in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:41,297 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/A is initiating minor compaction (all files) 2023-04-03 13:56:41,298 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/A in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:41,298 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/fc6faa16eff5470f8c6c9154f0c251c0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bae8a614b9f3433a8792115bf2c9908c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/9b1f9a9b457d462fbccc68ab3a6fa697, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/5b22fb7cc749400abc2e807a8a88fbb4] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=48.4 K 2023-04-03 13:56:41,298 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b2dcd0e86d844b4f81a6b848fa4f3844, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/44cbe38863a84eb3a68a7c6ac3bbe606, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/47fdfab908f14a2b800f564f74927420, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/ff612675c25a44278385f7e1c652c428] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=48.4 K 2023-04-03 13:56:41,298 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fc6faa16eff5470f8c6c9154f0c251c0, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680530195970 2023-04-03 13:56:41,298 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting b2dcd0e86d844b4f81a6b848fa4f3844, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680530195970 2023-04-03 13:56:41,298 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bae8a614b9f3433a8792115bf2c9908c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=988, earliestPutTs=1680530196592 2023-04-03 13:56:41,298 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 44cbe38863a84eb3a68a7c6ac3bbe606, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=988, earliestPutTs=1680530196592 2023-04-03 13:56:41,299 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 9b1f9a9b457d462fbccc68ab3a6fa697, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1680530197774 2023-04-03 13:56:41,299 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 47fdfab908f14a2b800f564f74927420, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1680530197774 2023-04-03 13:56:41,299 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5b22fb7cc749400abc2e807a8a88fbb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1038, earliestPutTs=1680530198903 2023-04-03 13:56:41,299 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting ff612675c25a44278385f7e1c652c428, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1038, earliestPutTs=1680530198903 2023-04-03 13:56:41,309 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#A#compaction#88 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:41,309 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#B#compaction#87 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:41,730 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/8450813e070449ed94c123a7efb253d1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/8450813e070449ed94c123a7efb253d1 2023-04-03 13:56:41,730 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/d5e0c75e96c94fc3a704a230193c362a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/d5e0c75e96c94fc3a704a230193c362a 2023-04-03 13:56:41,738 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/A of 3d9e5bf7331338c5e5778539b0a0c908 into d5e0c75e96c94fc3a704a230193c362a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:41,738 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/B of 3d9e5bf7331338c5e5778539b0a0c908 into 8450813e070449ed94c123a7efb253d1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:41,738 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:41,738 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:41,738 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/B, priority=12, startTime=1680530201296; duration=0sec 2023-04-03 13:56:41,738 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/A, priority=12, startTime=1680530201296; duration=0sec 2023-04-03 13:56:41,738 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:41,738 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:41,738 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:56:41,740 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:56:41,740 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 3d9e5bf7331338c5e5778539b0a0c908/C is initiating minor compaction (all files) 2023-04-03 13:56:41,740 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3d9e5bf7331338c5e5778539b0a0c908/C in TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:41,740 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/71c5fc93d1124feb9157bd1ef8cbcfde, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7888e498a6544920959b99ca7a96abe3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/b4c8605c1cda4c43ae602526060f4b39, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/11fbf4f0d13f46318f68c92db588bb36] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp, totalSize=48.4 K 2023-04-03 13:56:41,740 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 71c5fc93d1124feb9157bd1ef8cbcfde, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=979, earliestPutTs=1680530195970 2023-04-03 13:56:41,741 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 7888e498a6544920959b99ca7a96abe3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=988, earliestPutTs=1680530196592 2023-04-03 13:56:41,741 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting b4c8605c1cda4c43ae602526060f4b39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1028, earliestPutTs=1680530197774 2023-04-03 13:56:41,741 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 11fbf4f0d13f46318f68c92db588bb36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1038, earliestPutTs=1680530198903 2023-04-03 13:56:41,749 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 3d9e5bf7331338c5e5778539b0a0c908#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:41,759 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/cdaa5c4e3cf54015a25ea14c2c59f5ea as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/cdaa5c4e3cf54015a25ea14c2c59f5ea 2023-04-03 13:56:41,765 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3d9e5bf7331338c5e5778539b0a0c908/C of 3d9e5bf7331338c5e5778539b0a0c908 into cdaa5c4e3cf54015a25ea14c2c59f5ea(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:41,765 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:41,765 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908., storeName=3d9e5bf7331338c5e5778539b0a0c908/C, priority=12, startTime=1680530201296; duration=0sec 2023-04-03 13:56:41,765 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:46,383 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x142c8aea to 127.0.0.1:57081 2023-04-03 13:56:46,384 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:48,994 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e0ba002 to 127.0.0.1:57081 2023-04-03 13:56:48,994 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:49,002 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x136956fd to 127.0.0.1:57081 2023-04-03 13:56:49,002 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 325 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 201 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 131 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 170 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 121 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5100 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5072 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2160 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 6474 rows 2023-04-03 13:56:49,002 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2170 2023-04-03 13:56:49,003 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 6504 rows 2023-04-03 13:56:49,003 INFO [Listener at localhost.localdomain/43477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 13:56:49,003 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42d2ba41 to 127.0.0.1:57081 2023-04-03 13:56:49,003 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:56:49,007 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 13:56:49,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-03 13:56:49,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:49,018 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530209018"}]},"ts":"1680530209018"} 2023-04-03 13:56:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 13:56:49,019 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 13:56:49,026 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 13:56:49,028 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d9e5bf7331338c5e5778539b0a0c908, UNASSIGN}] 2023-04-03 13:56:49,029 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d9e5bf7331338c5e5778539b0a0c908, UNASSIGN 2023-04-03 13:56:49,029 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3d9e5bf7331338c5e5778539b0a0c908, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:49,030 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530209029"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530209029"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530209029"}]},"ts":"1680530209029"} 2023-04-03 13:56:49,032 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:56:49,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 13:56:49,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:49,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 3d9e5bf7331338c5e5778539b0a0c908, disabling compactions & flushes 2023-04-03 13:56:49,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:49,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:49,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. after waiting 0 ms 2023-04-03 13:56:49,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:49,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 3d9e5bf7331338c5e5778539b0a0c908 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 13:56:49,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=A 2023-04-03 13:56:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=B 2023-04-03 13:56:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3d9e5bf7331338c5e5778539b0a0c908, store=C 2023-04-03 13:56:49,194 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 13:56:49,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1048 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/935f91bc52464300b3601a4a002ff9b7 2023-04-03 13:56:49,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 13:56:50,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1048 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/308c5617948e4a38b21e88132899fb2e 2023-04-03 13:56:50,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 13:56:50,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1048 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/fa06dc74088a4867bd53d0907bef350a 2023-04-03 13:56:50,477 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/A/935f91bc52464300b3601a4a002ff9b7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/935f91bc52464300b3601a4a002ff9b7 2023-04-03 13:56:50,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/935f91bc52464300b3601a4a002ff9b7, entries=100, sequenceid=1048, filesize=9.5 K 2023-04-03 13:56:50,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/B/308c5617948e4a38b21e88132899fb2e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/308c5617948e4a38b21e88132899fb2e 2023-04-03 13:56:50,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/308c5617948e4a38b21e88132899fb2e, entries=100, sequenceid=1048, filesize=9.5 K 2023-04-03 13:56:50,491 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/.tmp/C/fa06dc74088a4867bd53d0907bef350a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/fa06dc74088a4867bd53d0907bef350a 2023-04-03 13:56:50,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/fa06dc74088a4867bd53d0907bef350a, entries=100, sequenceid=1048, filesize=9.5 K 2023-04-03 13:56:50,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 3d9e5bf7331338c5e5778539b0a0c908 in 1303ms, sequenceid=1048, compaction requested=false 2023-04-03 13:56:50,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b3ece2e8ce12486897d8132ba5fa8b7f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/8fdd8032155b45b991e5463301ab4535, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fb4e1acf0a68461d949b10d8789b793b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f0dd05b5651148b68464ee5c6b7ee921, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/c9b1ef0ff61c48be9459c844e756a8f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/48c05eeebc9d444d9497b817074021c0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/51c1394cd13141afac8803b091272de0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/50f3ed60b6b74353b1ab97b24f0ddd46, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/7b8b9ea48b9d4a87a94e87da1d138880, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/023e2ac38d834b7d9b6d42bc5add964d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/5d833c246320494f99e4e4a4add013e4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fbef6d14a01f41de85fde0a4040dfd92, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/07ec599ab54b4a74aeb18561e97e59ed, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9db906e8ae69422b95676560b55257e7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f77ec6e6f8dd42728e6ca93c2027dfbd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/46191ddc9e724015a9815e8bdacf264d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9850871386274c0da2fcf1f2a60d8ac0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/1fec538409734196af14df4010025d25, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/2ef65945d42849a4a4224a5bab111336, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4bbd82ad053f4fa893444e3d56ef99bc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/e78a2a4b2fb5422fbea62e629e553db6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b6895a9d38f547a58d18ede1b2dbcfe7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4ddaa006f9bf45c8866863f37c04d12b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/bd5543f2dae542eda68e02787e065db1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/745328121e9b4c418bd8435d81908a60, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b2dcd0e86d844b4f81a6b848fa4f3844, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/44cbe38863a84eb3a68a7c6ac3bbe606, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/47fdfab908f14a2b800f564f74927420, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/ff612675c25a44278385f7e1c652c428] to archive 2023-04-03 13:56:50,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:56:50,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b3ece2e8ce12486897d8132ba5fa8b7f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b3ece2e8ce12486897d8132ba5fa8b7f 2023-04-03 13:56:50,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/8fdd8032155b45b991e5463301ab4535 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/8fdd8032155b45b991e5463301ab4535 2023-04-03 13:56:50,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fb4e1acf0a68461d949b10d8789b793b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fb4e1acf0a68461d949b10d8789b793b 2023-04-03 13:56:50,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f0dd05b5651148b68464ee5c6b7ee921 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f0dd05b5651148b68464ee5c6b7ee921 2023-04-03 13:56:50,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/c9b1ef0ff61c48be9459c844e756a8f5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/c9b1ef0ff61c48be9459c844e756a8f5 2023-04-03 13:56:50,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/48c05eeebc9d444d9497b817074021c0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/48c05eeebc9d444d9497b817074021c0 2023-04-03 13:56:50,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/51c1394cd13141afac8803b091272de0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/51c1394cd13141afac8803b091272de0 2023-04-03 13:56:50,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/50f3ed60b6b74353b1ab97b24f0ddd46 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/50f3ed60b6b74353b1ab97b24f0ddd46 2023-04-03 13:56:50,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/7b8b9ea48b9d4a87a94e87da1d138880 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/7b8b9ea48b9d4a87a94e87da1d138880 2023-04-03 13:56:50,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/023e2ac38d834b7d9b6d42bc5add964d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/023e2ac38d834b7d9b6d42bc5add964d 2023-04-03 13:56:50,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/5d833c246320494f99e4e4a4add013e4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/5d833c246320494f99e4e4a4add013e4 2023-04-03 13:56:50,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fbef6d14a01f41de85fde0a4040dfd92 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/fbef6d14a01f41de85fde0a4040dfd92 2023-04-03 13:56:50,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/07ec599ab54b4a74aeb18561e97e59ed to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/07ec599ab54b4a74aeb18561e97e59ed 2023-04-03 13:56:50,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9db906e8ae69422b95676560b55257e7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9db906e8ae69422b95676560b55257e7 2023-04-03 13:56:50,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f77ec6e6f8dd42728e6ca93c2027dfbd to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/f77ec6e6f8dd42728e6ca93c2027dfbd 2023-04-03 13:56:50,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/46191ddc9e724015a9815e8bdacf264d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/46191ddc9e724015a9815e8bdacf264d 2023-04-03 13:56:50,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9850871386274c0da2fcf1f2a60d8ac0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/9850871386274c0da2fcf1f2a60d8ac0 2023-04-03 13:56:50,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/1fec538409734196af14df4010025d25 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/1fec538409734196af14df4010025d25 2023-04-03 13:56:50,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/2ef65945d42849a4a4224a5bab111336 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/2ef65945d42849a4a4224a5bab111336 2023-04-03 13:56:50,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4bbd82ad053f4fa893444e3d56ef99bc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4bbd82ad053f4fa893444e3d56ef99bc 2023-04-03 13:56:50,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/e78a2a4b2fb5422fbea62e629e553db6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/e78a2a4b2fb5422fbea62e629e553db6 2023-04-03 13:56:50,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b6895a9d38f547a58d18ede1b2dbcfe7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b6895a9d38f547a58d18ede1b2dbcfe7 2023-04-03 13:56:50,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4ddaa006f9bf45c8866863f37c04d12b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/4ddaa006f9bf45c8866863f37c04d12b 2023-04-03 13:56:50,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/bd5543f2dae542eda68e02787e065db1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/bd5543f2dae542eda68e02787e065db1 2023-04-03 13:56:50,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/745328121e9b4c418bd8435d81908a60 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/745328121e9b4c418bd8435d81908a60 2023-04-03 13:56:50,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b2dcd0e86d844b4f81a6b848fa4f3844 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/b2dcd0e86d844b4f81a6b848fa4f3844 2023-04-03 13:56:50,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/44cbe38863a84eb3a68a7c6ac3bbe606 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/44cbe38863a84eb3a68a7c6ac3bbe606 2023-04-03 13:56:50,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/47fdfab908f14a2b800f564f74927420 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/47fdfab908f14a2b800f564f74927420 2023-04-03 13:56:50,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/ff612675c25a44278385f7e1c652c428 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/ff612675c25a44278385f7e1c652c428 2023-04-03 13:56:50,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/188aca54624743e5997e1b60bd15441f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bc55fad51a144476b07013f7939ac091, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d84d9969421e4bf4b54337831c8b3d5f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e824049a43e94074ad89067bb9de8316, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/53e9362625b24d2d9744645218539d7a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/81d8176bc10246bfaaf50c87eb2364db, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/05b661bbc44c4bd8b833f5be80862759, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c5ac10808e64df49b1042e67158a09a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c2fd3021d7a4bbbbb8ab092d72c9556, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ec8b856bc98a4197a99dc7b3527574b8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bdd9b9748a8b4235a483f2878a1c001c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/491e549d36144b2b9944d6a6fb133c80, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d173dac546594a73b85c562d6733415a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2785eb0e58c0452bbbea7712b741dbde, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e97b3d322e0348d2b1aa6415bbe283c7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/cb9589775180480d872f0ca114c15b8e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/a9c92dd2218f43e38d39037d828dd074, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/40dcd617c42e497990afda6eb7c8cb14, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/601a7bdb4d34413c8dfa721c6fa90f07, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/351b09f1718d40b6aa00cdf380b0cfcb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/8c0b3ef947c9480ab98d32d95eace0ba, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e50d94b6efb64c60902812ddd8cb0e42, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ad96a5898893478da09f3f082a72c74e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/68c5ca72d8a94bc698ed2f4460959154, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/fc6faa16eff5470f8c6c9154f0c251c0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/c35be200f9934bc587b2bf2bd1bd0c00, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bae8a614b9f3433a8792115bf2c9908c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/9b1f9a9b457d462fbccc68ab3a6fa697, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/5b22fb7cc749400abc2e807a8a88fbb4] to archive 2023-04-03 13:56:50,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:56:50,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/188aca54624743e5997e1b60bd15441f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/188aca54624743e5997e1b60bd15441f 2023-04-03 13:56:50,585 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bc55fad51a144476b07013f7939ac091 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bc55fad51a144476b07013f7939ac091 2023-04-03 13:56:50,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d84d9969421e4bf4b54337831c8b3d5f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d84d9969421e4bf4b54337831c8b3d5f 2023-04-03 13:56:50,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e824049a43e94074ad89067bb9de8316 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e824049a43e94074ad89067bb9de8316 2023-04-03 13:56:50,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/53e9362625b24d2d9744645218539d7a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/53e9362625b24d2d9744645218539d7a 2023-04-03 13:56:50,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/81d8176bc10246bfaaf50c87eb2364db to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/81d8176bc10246bfaaf50c87eb2364db 2023-04-03 13:56:50,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/05b661bbc44c4bd8b833f5be80862759 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/05b661bbc44c4bd8b833f5be80862759 2023-04-03 13:56:50,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c5ac10808e64df49b1042e67158a09a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c5ac10808e64df49b1042e67158a09a 2023-04-03 13:56:50,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c2fd3021d7a4bbbbb8ab092d72c9556 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2c2fd3021d7a4bbbbb8ab092d72c9556 2023-04-03 13:56:50,597 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ec8b856bc98a4197a99dc7b3527574b8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ec8b856bc98a4197a99dc7b3527574b8 2023-04-03 13:56:50,598 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bdd9b9748a8b4235a483f2878a1c001c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bdd9b9748a8b4235a483f2878a1c001c 2023-04-03 13:56:50,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/491e549d36144b2b9944d6a6fb133c80 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/491e549d36144b2b9944d6a6fb133c80 2023-04-03 13:56:50,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d173dac546594a73b85c562d6733415a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/d173dac546594a73b85c562d6733415a 2023-04-03 13:56:50,602 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2785eb0e58c0452bbbea7712b741dbde to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/2785eb0e58c0452bbbea7712b741dbde 2023-04-03 13:56:50,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e97b3d322e0348d2b1aa6415bbe283c7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e97b3d322e0348d2b1aa6415bbe283c7 2023-04-03 13:56:50,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/cb9589775180480d872f0ca114c15b8e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/cb9589775180480d872f0ca114c15b8e 2023-04-03 13:56:50,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/a9c92dd2218f43e38d39037d828dd074 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/a9c92dd2218f43e38d39037d828dd074 2023-04-03 13:56:50,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/40dcd617c42e497990afda6eb7c8cb14 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/40dcd617c42e497990afda6eb7c8cb14 2023-04-03 13:56:50,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/601a7bdb4d34413c8dfa721c6fa90f07 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/601a7bdb4d34413c8dfa721c6fa90f07 2023-04-03 13:56:50,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/351b09f1718d40b6aa00cdf380b0cfcb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/351b09f1718d40b6aa00cdf380b0cfcb 2023-04-03 13:56:50,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/8c0b3ef947c9480ab98d32d95eace0ba to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/8c0b3ef947c9480ab98d32d95eace0ba 2023-04-03 13:56:50,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e50d94b6efb64c60902812ddd8cb0e42 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/e50d94b6efb64c60902812ddd8cb0e42 2023-04-03 13:56:50,615 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ad96a5898893478da09f3f082a72c74e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/ad96a5898893478da09f3f082a72c74e 2023-04-03 13:56:50,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/68c5ca72d8a94bc698ed2f4460959154 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/68c5ca72d8a94bc698ed2f4460959154 2023-04-03 13:56:50,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/fc6faa16eff5470f8c6c9154f0c251c0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/fc6faa16eff5470f8c6c9154f0c251c0 2023-04-03 13:56:50,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/c35be200f9934bc587b2bf2bd1bd0c00 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/c35be200f9934bc587b2bf2bd1bd0c00 2023-04-03 13:56:50,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bae8a614b9f3433a8792115bf2c9908c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/bae8a614b9f3433a8792115bf2c9908c 2023-04-03 13:56:50,622 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/9b1f9a9b457d462fbccc68ab3a6fa697 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/9b1f9a9b457d462fbccc68ab3a6fa697 2023-04-03 13:56:50,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/5b22fb7cc749400abc2e807a8a88fbb4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/5b22fb7cc749400abc2e807a8a88fbb4 2023-04-03 13:56:50,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d8ee97500dcb42f18c6c814859a3ea61, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/0a7ae797162147c48bbc4c1fd06a538f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/33eb10cd1905439b8a3b296c5293837c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/34258749c1a743e2b36adae996de0cce, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/395aa8afcb554baf9c449c349c6b1f73, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e677ecd730b0422eaa72c7ac5e4d96fc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/45eb3e8939a8464583ff378842be6ea4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/f938c1d771154f9989c86768efa612a5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/24177b567d1f4bce8477203eb49c3422, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/de8c4d40ce6d49069fd4a0329c3f8541, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/6f408cd02f394852ab6642895d718d9b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ed587517cc5744108975c8a3ad1e18c0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a020494a82d24e058eae59b6c714e1ac, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d130cfc165c94fed8574c613af08675e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/1e3d2e5565774c33a809afadc9d43648, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/8f06a3e671b34b2899549766e6bd91b2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/569ee40e670d486cadb611c6d7f5646c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7088fdeb21014ee7b927a6c5d53afa08, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/3eb20929a5ed4edcbd1471811ef48b12, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/833dd31f23b6428f84dc49025e597056, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e824d66381844187963f6280579df336, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ad85a877a56e4dcea389c29c67318f96, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/36eb8b5cdd184faba29fc9083867924d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/4dbc2d4b24c74bfdbefe67d0b1e59cd4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/71c5fc93d1124feb9157bd1ef8cbcfde, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a1e6b874500046bd86f7a631910629c4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7888e498a6544920959b99ca7a96abe3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/b4c8605c1cda4c43ae602526060f4b39, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/11fbf4f0d13f46318f68c92db588bb36] to archive 2023-04-03 13:56:50,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:56:50,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d8ee97500dcb42f18c6c814859a3ea61 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d8ee97500dcb42f18c6c814859a3ea61 2023-04-03 13:56:50,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/0a7ae797162147c48bbc4c1fd06a538f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/0a7ae797162147c48bbc4c1fd06a538f 2023-04-03 13:56:50,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/33eb10cd1905439b8a3b296c5293837c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/33eb10cd1905439b8a3b296c5293837c 2023-04-03 13:56:50,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/34258749c1a743e2b36adae996de0cce to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/34258749c1a743e2b36adae996de0cce 2023-04-03 13:56:50,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/395aa8afcb554baf9c449c349c6b1f73 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/395aa8afcb554baf9c449c349c6b1f73 2023-04-03 13:56:50,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e677ecd730b0422eaa72c7ac5e4d96fc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e677ecd730b0422eaa72c7ac5e4d96fc 2023-04-03 13:56:50,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/45eb3e8939a8464583ff378842be6ea4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/45eb3e8939a8464583ff378842be6ea4 2023-04-03 13:56:50,639 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/f938c1d771154f9989c86768efa612a5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/f938c1d771154f9989c86768efa612a5 2023-04-03 13:56:50,640 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/24177b567d1f4bce8477203eb49c3422 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/24177b567d1f4bce8477203eb49c3422 2023-04-03 13:56:50,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/de8c4d40ce6d49069fd4a0329c3f8541 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/de8c4d40ce6d49069fd4a0329c3f8541 2023-04-03 13:56:50,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/6f408cd02f394852ab6642895d718d9b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/6f408cd02f394852ab6642895d718d9b 2023-04-03 13:56:50,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ed587517cc5744108975c8a3ad1e18c0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ed587517cc5744108975c8a3ad1e18c0 2023-04-03 13:56:50,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a020494a82d24e058eae59b6c714e1ac to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a020494a82d24e058eae59b6c714e1ac 2023-04-03 13:56:50,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d130cfc165c94fed8574c613af08675e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/d130cfc165c94fed8574c613af08675e 2023-04-03 13:56:50,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/1e3d2e5565774c33a809afadc9d43648 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/1e3d2e5565774c33a809afadc9d43648 2023-04-03 13:56:50,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/8f06a3e671b34b2899549766e6bd91b2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/8f06a3e671b34b2899549766e6bd91b2 2023-04-03 13:56:50,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/569ee40e670d486cadb611c6d7f5646c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/569ee40e670d486cadb611c6d7f5646c 2023-04-03 13:56:50,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7088fdeb21014ee7b927a6c5d53afa08 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7088fdeb21014ee7b927a6c5d53afa08 2023-04-03 13:56:50,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/3eb20929a5ed4edcbd1471811ef48b12 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/3eb20929a5ed4edcbd1471811ef48b12 2023-04-03 13:56:50,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/833dd31f23b6428f84dc49025e597056 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/833dd31f23b6428f84dc49025e597056 2023-04-03 13:56:50,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e824d66381844187963f6280579df336 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/e824d66381844187963f6280579df336 2023-04-03 13:56:50,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ad85a877a56e4dcea389c29c67318f96 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/ad85a877a56e4dcea389c29c67318f96 2023-04-03 13:56:50,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/36eb8b5cdd184faba29fc9083867924d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/36eb8b5cdd184faba29fc9083867924d 2023-04-03 13:56:50,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/4dbc2d4b24c74bfdbefe67d0b1e59cd4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/4dbc2d4b24c74bfdbefe67d0b1e59cd4 2023-04-03 13:56:50,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/71c5fc93d1124feb9157bd1ef8cbcfde to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/71c5fc93d1124feb9157bd1ef8cbcfde 2023-04-03 13:56:50,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a1e6b874500046bd86f7a631910629c4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/a1e6b874500046bd86f7a631910629c4 2023-04-03 13:56:50,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7888e498a6544920959b99ca7a96abe3 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/7888e498a6544920959b99ca7a96abe3 2023-04-03 13:56:50,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/b4c8605c1cda4c43ae602526060f4b39 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/b4c8605c1cda4c43ae602526060f4b39 2023-04-03 13:56:50,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/11fbf4f0d13f46318f68c92db588bb36 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/11fbf4f0d13f46318f68c92db588bb36 2023-04-03 13:56:50,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/recovered.edits/1051.seqid, newMaxSeqId=1051, maxSeqId=1 2023-04-03 13:56:50,669 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908. 2023-04-03 13:56:50,669 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 3d9e5bf7331338c5e5778539b0a0c908: 2023-04-03 13:56:50,671 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:50,671 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=3d9e5bf7331338c5e5778539b0a0c908, regionState=CLOSED 2023-04-03 13:56:50,671 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530210671"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530210671"}]},"ts":"1680530210671"} 2023-04-03 13:56:50,674 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-03 13:56:50,674 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 3d9e5bf7331338c5e5778539b0a0c908, server=jenkins-hbase20.apache.org,39695,1680530174856 in 1.6410 sec 2023-04-03 13:56:50,675 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-03 13:56:50,675 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3d9e5bf7331338c5e5778539b0a0c908, UNASSIGN in 1.6460 sec 2023-04-03 13:56:50,676 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530210676"}]},"ts":"1680530210676"} 2023-04-03 13:56:50,677 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 13:56:50,678 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 13:56:50,680 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6670 sec 2023-04-03 13:56:51,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 13:56:51,128 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-03 13:56:51,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-03 13:56:51,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:51,146 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:51,147 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:51,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-03 13:56:51,152 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:51,156 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/recovered.edits] 2023-04-03 13:56:51,160 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/935f91bc52464300b3601a4a002ff9b7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/935f91bc52464300b3601a4a002ff9b7 2023-04-03 13:56:51,162 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/d5e0c75e96c94fc3a704a230193c362a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/A/d5e0c75e96c94fc3a704a230193c362a 2023-04-03 13:56:51,165 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/308c5617948e4a38b21e88132899fb2e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/308c5617948e4a38b21e88132899fb2e 2023-04-03 13:56:51,167 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/8450813e070449ed94c123a7efb253d1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/B/8450813e070449ed94c123a7efb253d1 2023-04-03 13:56:51,170 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/cdaa5c4e3cf54015a25ea14c2c59f5ea to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/cdaa5c4e3cf54015a25ea14c2c59f5ea 2023-04-03 13:56:51,172 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/fa06dc74088a4867bd53d0907bef350a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/C/fa06dc74088a4867bd53d0907bef350a 2023-04-03 13:56:51,177 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/recovered.edits/1051.seqid to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908/recovered.edits/1051.seqid 2023-04-03 13:56:51,178 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/3d9e5bf7331338c5e5778539b0a0c908 2023-04-03 13:56:51,178 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:56:51,182 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:51,210 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 13:56:51,246 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 13:56:51,248 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:51,248 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 13:56:51,248 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680530211248"}]},"ts":"9223372036854775807"} 2023-04-03 13:56:51,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-03 13:56:51,250 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 13:56:51,250 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3d9e5bf7331338c5e5778539b0a0c908, NAME => 'TestAcidGuarantees,,1680530178393.3d9e5bf7331338c5e5778539b0a0c908.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 13:56:51,250 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 13:56:51,250 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680530211250"}]},"ts":"9223372036854775807"} 2023-04-03 13:56:51,254 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 13:56:51,261 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:51,262 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 121 msec 2023-04-03 13:56:51,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-03 13:56:51,451 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-03 13:56:51,478 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:39695-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e0d66be-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e0d66be-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e0d66be-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1574154525_15 at /127.0.0.1:55710 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x3e0d66be-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=548 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=273 (was 172) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=2919 (was 3493) 2023-04-03 13:56:51,486 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=548, MaxFileDescriptor=60000, SystemLoadAverage=273, ProcessCount=166, AvailableMemoryMB=2919 2023-04-03 13:56:51,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:56:51,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:56:51,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:51,491 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 13:56:51,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-03 13:56:51,492 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 13:56:51,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 13:56:51,494 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:51,494 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 empty. 2023-04-03 13:56:51,495 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:51,495 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:56:51,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 13:56:51,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 13:56:51,907 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:56:51,914 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 67b1fdef2f316cba9760895849ff9624, NAME => 'TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp 2023-04-03 13:56:52,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 13:56:52,322 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:52,323 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 67b1fdef2f316cba9760895849ff9624, disabling compactions & flushes 2023-04-03 13:56:52,323 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:52,323 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:52,323 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. after waiting 0 ms 2023-04-03 13:56:52,323 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:52,323 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:52,324 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:52,331 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 13:56:52,334 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530212333"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530212333"}]},"ts":"1680530212333"} 2023-04-03 13:56:52,335 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 13:56:52,336 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 13:56:52,336 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530212336"}]},"ts":"1680530212336"} 2023-04-03 13:56:52,337 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 13:56:52,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, ASSIGN}] 2023-04-03 13:56:52,341 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, ASSIGN 2023-04-03 13:56:52,342 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=false 2023-04-03 13:56:52,494 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=67b1fdef2f316cba9760895849ff9624, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:52,494 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530212493"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530212493"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530212493"}]},"ts":"1680530212493"} 2023-04-03 13:56:52,497 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:56:52,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 13:56:52,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:52,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 67b1fdef2f316cba9760895849ff9624, NAME => 'TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:56:52,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:52,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,660 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,662 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,663 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A 2023-04-03 13:56:52,663 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A 2023-04-03 13:56:52,664 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:52,664 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67b1fdef2f316cba9760895849ff9624 columnFamilyName A 2023-04-03 13:56:52,664 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(310): Store=67b1fdef2f316cba9760895849ff9624/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:52,665 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,666 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B 2023-04-03 13:56:52,666 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B 2023-04-03 13:56:52,666 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:52,666 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67b1fdef2f316cba9760895849ff9624 columnFamilyName B 2023-04-03 13:56:52,667 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(310): Store=67b1fdef2f316cba9760895849ff9624/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:52,667 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,668 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C 2023-04-03 13:56:52,668 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C 2023-04-03 13:56:52,668 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:52,668 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67b1fdef2f316cba9760895849ff9624 columnFamilyName C 2023-04-03 13:56:52,668 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(310): Store=67b1fdef2f316cba9760895849ff9624/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:52,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:52,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,669 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:56:52,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:52,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:56:52,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 67b1fdef2f316cba9760895849ff9624; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64394094, jitterRate=-0.0404532253742218}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:56:52,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:52,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., pid=18, masterSystemTime=1680530212652 2023-04-03 13:56:52,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:52,677 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:52,677 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=67b1fdef2f316cba9760895849ff9624, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:52,678 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530212677"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530212677"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530212677"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530212677"}]},"ts":"1680530212677"} 2023-04-03 13:56:52,681 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-03 13:56:52,681 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 in 182 msec 2023-04-03 13:56:52,682 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-03 13:56:52,683 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, ASSIGN in 342 msec 2023-04-03 13:56:52,683 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 13:56:52,683 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530212683"}]},"ts":"1680530212683"} 2023-04-03 13:56:52,684 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 13:56:52,686 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 13:56:52,687 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1970 sec 2023-04-03 13:56:53,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 13:56:53,603 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-03 13:56:53,604 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x35bef85d to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:53,612 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3105e416, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:53,614 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:53,619 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:53,621 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 13:56:53,624 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49682, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 13:56:53,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:56:53,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:56:53,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 13:56:54,053 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 13:56:54,056 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:56:54,056 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 13:56:54,063 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 13:56:54,075 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, REOPEN/MOVE}] 2023-04-03 13:56:54,076 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, REOPEN/MOVE 2023-04-03 13:56:54,077 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=67b1fdef2f316cba9760895849ff9624, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,077 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530214077"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530214077"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530214077"}]},"ts":"1680530214077"} 2023-04-03 13:56:54,078 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:56:54,230 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 67b1fdef2f316cba9760895849ff9624, disabling compactions & flushes 2023-04-03 13:56:54,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. after waiting 0 ms 2023-04-03 13:56:54,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 13:56:54,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:54,241 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 67b1fdef2f316cba9760895849ff9624 to self. 2023-04-03 13:56:54,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,243 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=67b1fdef2f316cba9760895849ff9624, regionState=CLOSED 2023-04-03 13:56:54,243 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530214243"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530214243"}]},"ts":"1680530214243"} 2023-04-03 13:56:54,246 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-03 13:56:54,246 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 in 166 msec 2023-04-03 13:56:54,247 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=true 2023-04-03 13:56:54,397 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=67b1fdef2f316cba9760895849ff9624, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,398 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530214397"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530214397"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530214397"}]},"ts":"1680530214397"} 2023-04-03 13:56:54,401 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:56:54,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 67b1fdef2f316cba9760895849ff9624, NAME => 'TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:56:54,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:56:54,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,568 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,569 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A 2023-04-03 13:56:54,569 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A 2023-04-03 13:56:54,570 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:54,573 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67b1fdef2f316cba9760895849ff9624 columnFamilyName A 2023-04-03 13:56:54,575 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(310): Store=67b1fdef2f316cba9760895849ff9624/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:54,576 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,578 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B 2023-04-03 13:56:54,578 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B 2023-04-03 13:56:54,578 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:54,578 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67b1fdef2f316cba9760895849ff9624 columnFamilyName B 2023-04-03 13:56:54,579 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(310): Store=67b1fdef2f316cba9760895849ff9624/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:54,579 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,580 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C 2023-04-03 13:56:54,580 DEBUG [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C 2023-04-03 13:56:54,580 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:56:54,580 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67b1fdef2f316cba9760895849ff9624 columnFamilyName C 2023-04-03 13:56:54,581 INFO [StoreOpener-67b1fdef2f316cba9760895849ff9624-1] regionserver.HStore(310): Store=67b1fdef2f316cba9760895849ff9624/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:56:54,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:56:54,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 67b1fdef2f316cba9760895849ff9624; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73319693, jitterRate=0.09254856407642365}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:56:54,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:54,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., pid=23, masterSystemTime=1680530214554 2023-04-03 13:56:54,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,589 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=67b1fdef2f316cba9760895849ff9624, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,589 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530214589"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530214589"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530214589"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530214589"}]},"ts":"1680530214589"} 2023-04-03 13:56:54,592 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-03 13:56:54,592 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 in 189 msec 2023-04-03 13:56:54,593 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-03 13:56:54,593 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, REOPEN/MOVE in 517 msec 2023-04-03 13:56:54,597 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-03 13:56:54,597 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 533 msec 2023-04-03 13:56:54,599 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 963 msec 2023-04-03 13:56:54,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-03 13:56:54,609 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x071f9a8a to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,613 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ef5b30d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,614 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x430c7474 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,619 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b75e45b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,619 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x2d781def to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,623 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@df763e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,624 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x5649f410 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,633 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6017755a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,634 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x7acc1387 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,644 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@633c81ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,644 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x61ad5d1c to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,652 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@661b21c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,653 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x486fba63 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,660 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43a14bcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,661 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x50feca84 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,668 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6577ce5a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,669 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x0bd79dff to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:56:54,675 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d5bdceb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:56:54,686 DEBUG [hconnection-0x52ec2488-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,686 DEBUG [hconnection-0x393c666e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,686 DEBUG [hconnection-0x7f79998a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,686 DEBUG [hconnection-0x5b967cff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,686 DEBUG [hconnection-0x7c98aba3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,686 DEBUG [hconnection-0x76c6a5f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,686 DEBUG [hconnection-0x14b8e50d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:54,688 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41038, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,688 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41040, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,689 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,694 DEBUG [hconnection-0x489cbcf9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,694 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41044, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,695 DEBUG [hconnection-0xa5785e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:56:54,694 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41042, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,697 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41076, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,698 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41082, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,700 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,708 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:56:54,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:54,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:54,712 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:54,712 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:54,713 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:54,713 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:54,713 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,713 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:54,718 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,718 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:56:54,719 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:54,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:54,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:56:54,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:54,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:56:54,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,719 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:56:54,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:54,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:54,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:54,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:54,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:54,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:54,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530274733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530274734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530214730, max timestamp=1680530214736Num uniques 859; 2023-04-03 13:56:54,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:54,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530214730, max timestamp=1680530214736Num uniques 859; 2023-04-03 13:56:54,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:54,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530214730, max timestamp=1680530214736Num uniques 859; 2023-04-03 13:56:54,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:56:54,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530274759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530274760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530274761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,775 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:54,775 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:54,776 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,776 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:54,776 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:54,776 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,776 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:54,776 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:54,776 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:54,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:54,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:54,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:54,781 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:54,781 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,781 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:54,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:54,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@585424f3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:54,782 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:54,782 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,784 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,784 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:54,784 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:54,784 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:54,784 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,784 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:54,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,785 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:54,785 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:54,785 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:54,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:54,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:54,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,792 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040368789b4fdabc4f31945e7edfe0b22845 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040368789b4fdabc4f31945e7edfe0b22845 2023-04-03 13:56:54,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:54,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:54,793 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:54,793 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:54,793 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:54,793 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:54,793 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:54,796 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,796 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:54,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,796 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:54,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:54,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:54,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:54,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:54,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,800 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,800 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:54,800 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:54,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:54,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:54,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:54,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:54,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:54,800 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:54,800 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:54,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:54,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:54,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:54,818 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/5d66fc1b311f40b7aaab3da0da8a56a0 2023-04-03 13:56:54,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530274839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530274839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/2515b541ee814bd081832cbdd9fb77d0 2023-04-03 13:56:54,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530274863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530274863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:54,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530274863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,900 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:54,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:54,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:54,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:54,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:54,908 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:54,908 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:54,908 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:54,908 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:54,908 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,908 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,909 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:54,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:54,909 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:54,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:54,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:54,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:54,911 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,911 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:54,911 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:54,911 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:54,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:54,913 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,913 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:54,914 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,914 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:54,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:54,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d95e4d4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:54,915 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:54,915 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,916 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:54,916 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:54,916 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:54,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:54,918 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:54,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:54,921 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:54,921 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:54,921 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:54,921 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:54,921 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:54,925 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,926 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:54,926 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:54,926 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,928 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:54,930 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:54,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:54,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:54,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:54,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:54,930 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:54,930 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:54,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:54,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:55,030 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:55,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:55,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:55,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530275041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530275041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:55,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:55,050 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:55,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:55,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:55,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,060 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:55,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530275065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530275066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530275066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:55,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:55,070 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:55,070 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,070 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:55,080 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:55,080 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,080 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:55,080 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:55,081 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:55,081 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:55,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:55,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,083 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:55,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b29c868[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:55,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:55,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,086 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:55,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:55,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:55,087 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,087 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:55,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:55,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:55,088 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:55,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:55,092 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:55,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:55,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:55,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:55,093 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:55,101 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,101 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,101 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,107 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:55,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:55,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,110 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:55,110 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:55,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:55,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,110 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,210 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:55,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:55,218 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:55,218 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,218 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:55,218 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:55,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,219 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:55,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:55,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:55,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:55,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:55,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:55,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:55,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:55,229 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,229 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:55,229 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:55,229 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:55,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:55,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:55,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:55,234 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:55,236 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:55,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4e2481[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:55,236 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,236 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,237 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:55,237 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:55,237 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:55,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:55,238 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,238 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:55,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:55,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:55,243 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:55,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:55,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:55,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:55,243 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:55,252 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,252 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,252 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,253 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,255 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:55,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:55,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,258 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:55,258 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,258 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/d9c3bff28b114e94be1258b7214482d2 2023-04-03 13:56:55,308 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/5d66fc1b311f40b7aaab3da0da8a56a0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5d66fc1b311f40b7aaab3da0da8a56a0 2023-04-03 13:56:55,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5d66fc1b311f40b7aaab3da0da8a56a0, entries=200, sequenceid=16, filesize=31.9 K 2023-04-03 13:56:55,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/2515b541ee814bd081832cbdd9fb77d0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2515b541ee814bd081832cbdd9fb77d0 2023-04-03 13:56:55,327 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2515b541ee814bd081832cbdd9fb77d0, entries=150, sequenceid=16, filesize=11.6 K 2023-04-03 13:56:55,328 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/d9c3bff28b114e94be1258b7214482d2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d9c3bff28b114e94be1258b7214482d2 2023-04-03 13:56:55,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530275344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530275345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d9c3bff28b114e94be1258b7214482d2, entries=150, sequenceid=16, filesize=11.6 K 2023-04-03 13:56:55,356 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 67b1fdef2f316cba9760895849ff9624 in 638ms, sequenceid=16, compaction requested=false 2023-04-03 13:56:55,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:55,358 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:55,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:55,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-03 13:56:55,369 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:56:55,369 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:55,369 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:56:55,369 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:55,369 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:56:55,369 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:56:55,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:55,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:55,370 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:55,370 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,370 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:55,370 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:55,371 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,371 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,372 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,373 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,373 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:55,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:55,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:55,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530275381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530275382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530275383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:55,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:55,384 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:55,384 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,384 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:55,386 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:55,386 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:55,386 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,386 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:55,386 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:55,386 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,386 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:55,386 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:55,386 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,387 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:55,387 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:55,389 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:55,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:55,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7344ea87[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:55,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:55,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:55,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:55,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:55,392 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,392 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:55,393 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:55,393 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:55,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:55,395 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:55,396 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:55,396 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:55,396 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:55,396 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:55,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:55,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,400 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,400 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:55,400 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:55,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,427 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403fb48e06eac404b79a047ea6150df0649 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fb48e06eac404b79a047ea6150df0649 2023-04-03 13:56:55,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530275484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530275484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530275484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,500 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:55,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:55,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:55,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:55,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:55,505 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:55,505 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,505 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:55,505 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:55,505 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,505 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,506 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:55,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:55,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:55,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:55,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:55,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:55,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:55,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:55,510 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,510 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:55,510 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:55,510 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:55,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:55,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:55,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:55,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,515 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:55,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:55,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2125137f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:55,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,525 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:55,526 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:55,526 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:55,526 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:55,532 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:55,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,532 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,532 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:55,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:55,534 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:55,534 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:55,534 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:55,534 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:55,534 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:55,542 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,542 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,542 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,543 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:55,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:55,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:55,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,549 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:55,549 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,649 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:55,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:55,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:55,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:55,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:55,653 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:55,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:55,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:55,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,654 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,654 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,654 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:55,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:55,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:55,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:55,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:55,655 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:55,655 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,655 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:55,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:55,663 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:55,663 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:55,663 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:55,663 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:55,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:55,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:55,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:55,667 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,667 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,668 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,668 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:55,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44aae5bb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:55,676 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:55,676 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,677 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:55,677 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:55,677 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:55,677 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:55,678 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:55,678 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,678 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:55,678 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:55,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:55,681 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:55,681 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:55,681 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:55,681 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:55,681 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:55,683 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,683 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,685 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530275687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530275687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530275688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,693 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:55,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:55,694 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:55,694 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:55,794 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:55,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:55,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:55,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:55,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:55,797 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:55,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:55,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:55,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:55,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:55,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:55,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:55,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:55,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:55,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:55,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:55,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:55,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:55,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:55,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:55,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:55,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:55,804 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:55,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fbbab83[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:55,804 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,805 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:55,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:55,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:55,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:55,805 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:55,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:55,807 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:55,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:55,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:55,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:55,807 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:55,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,811 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:55,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:55,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,811 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:55,812 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:55,840 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/f94d62b93f284513b601b2f403466fe5 2023-04-03 13:56:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530275852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530275853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/fb8ff36826a54dfa9bce78d577aec580 2023-04-03 13:56:55,887 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/65a165921a274cc888a3adaa2d2387a3 2023-04-03 13:56:55,892 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/f94d62b93f284513b601b2f403466fe5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f94d62b93f284513b601b2f403466fe5 2023-04-03 13:56:55,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f94d62b93f284513b601b2f403466fe5, entries=150, sequenceid=52, filesize=25.1 K 2023-04-03 13:56:55,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/fb8ff36826a54dfa9bce78d577aec580 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fb8ff36826a54dfa9bce78d577aec580 2023-04-03 13:56:55,912 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:55,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:55,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fb8ff36826a54dfa9bce78d577aec580, entries=150, sequenceid=52, filesize=11.6 K 2023-04-03 13:56:55,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:55,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/65a165921a274cc888a3adaa2d2387a3 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/65a165921a274cc888a3adaa2d2387a3 2023-04-03 13:56:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:55,916 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:55,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:55,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:55,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,917 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,918 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,918 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:55,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:55,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:55,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:55,919 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:55,919 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:55,919 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:55,919 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:55,920 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:55,920 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,920 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:55,920 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:55,920 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:55,920 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:55,920 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:55,920 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,920 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:55,920 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:55,921 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:55,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bff6e21[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:55,923 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:55,923 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:55,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:55,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:55,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:55,925 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:55,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:55,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:55,929 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:55,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:55,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:55,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:55,929 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:55,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/65a165921a274cc888a3adaa2d2387a3, entries=150, sequenceid=52, filesize=11.6 K 2023-04-03 13:56:55,930 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 67b1fdef2f316cba9760895849ff9624 in 561ms, sequenceid=52, compaction requested=false 2023-04-03 13:56:55,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:55,930 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:55,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:55,932 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:55,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:55,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:55,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:55,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:55,933 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:55,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:55,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:55,934 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:55,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:55,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:55,934 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:55,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:55,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:55,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:55,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 13:56:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:56:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:56:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:56:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:56,009 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040363511c8dc9244565b25ca3b85c7e29c6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040363511c8dc9244565b25ca3b85c7e29c6 2023-04-03 13:56:56,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530276009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530276011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530276012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,034 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:56,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:56,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:56,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:56,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:56,037 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:56,037 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,037 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:56,037 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:56,037 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,037 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,038 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:56,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:56,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:56,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:56,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:56,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:56,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:56,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,041 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:56,041 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:56,041 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:56,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:56,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:56,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:56,042 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:56,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:56,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c836ca[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:56,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,044 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,044 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:56,044 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:56,044 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:56,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:56,045 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:56,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:56,047 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:56,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:56,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:56,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:56,047 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:56,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,049 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,049 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:56,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:56,053 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,053 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:56,054 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:56,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,115 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530276113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530276113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530276113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,154 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:56,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:56,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:56,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:56,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:56,157 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:56,157 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,157 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:56,157 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:56,157 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,158 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,158 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,158 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,158 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,158 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:56,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:56,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:56,160 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,160 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:56,160 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:56,160 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:56,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:56,161 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,161 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,161 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,161 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:56,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715c9015[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:56,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,163 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,163 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:56,163 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:56,163 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:56,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:56,164 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:56,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:56,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:56,166 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:56,166 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:56,166 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:56,166 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:56,166 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:56,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,168 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,169 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:56,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,170 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,170 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,213 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 13:56:56,271 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:56,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:56,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:56,278 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:56,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:56,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:56,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,280 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:56,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:56,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:56,281 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:56,281 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,281 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:56,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:56,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:56,282 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,282 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:56,282 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:56,282 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:56,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:56,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:56,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:56,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,284 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,284 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:56,285 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:56,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1359da0e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:56,285 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,286 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:56,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,286 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:56,286 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:56,286 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:56,288 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:56,288 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,288 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:56,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:56,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:56,291 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:56,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:56,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:56,291 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:56,291 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:56,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,294 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:56,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,298 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,298 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:56,298 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:56,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530276318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530276318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530276319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,398 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:56,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:56,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:56,402 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:56,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:56,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:56,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,403 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,403 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,403 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,403 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:56,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:56,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:56,404 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:56,404 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,404 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:56,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:56,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:56,405 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,405 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:56,405 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:56,405 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:56,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:56,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:56,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:56,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3473022[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:56,408 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:56,408 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,409 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:56,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:56,409 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,409 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:56,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:56,410 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,410 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:56,410 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,410 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:56,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:56,413 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:56,413 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:56,413 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:56,413 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:56,413 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:56,414 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,414 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,414 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,415 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,417 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,418 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=66, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/5c131959cff6492b975bb82a31ec10be 2023-04-03 13:56:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,421 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,421 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,422 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:56,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:56,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,422 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:56,422 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:56,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:56,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/e8960e1c3f2646819af9999b1831accf 2023-04-03 13:56:56,522 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:56,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:56,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:56,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:56,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:56,525 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:56,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:56,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:56,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,526 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,527 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:56,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:56,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:56,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:56,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:56,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:56,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:56,528 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,528 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:56,528 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:56,528 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:56,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:56,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:56,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:56,529 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,529 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,529 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,529 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:56,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209a38e6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:56,530 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:56,530 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,531 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,531 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,531 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:56,531 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:56,531 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:56,531 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:56,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,532 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,532 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,532 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:56,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:56,534 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:56,534 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:56,534 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:56,534 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:56,534 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:56,535 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,535 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,535 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,535 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,536 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,538 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:56,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,538 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:56,538 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:56,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530276620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530276621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530276624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,639 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:56,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:56,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:56,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:56,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:56,642 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:56,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:56,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:56,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:56,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:56,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:56,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:56,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:56,644 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:56,644 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:56,644 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:56,644 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:56,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:56,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,645 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,646 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,646 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:56,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3553d8d8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:56,647 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:56,647 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,647 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:56,648 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:56,648 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:56,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:56,648 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,648 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,648 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,648 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:56,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:56,651 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:56,651 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:56,651 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:56,651 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:56,652 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:56,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,654 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,655 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:56,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:56,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,656 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,656 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,656 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,656 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,656 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,756 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:56,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:56,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:56,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:56,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:56,759 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:56,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:56,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:56,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:56,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:56,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:56,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:56,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:56,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:56,762 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,762 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:56,762 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:56,762 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:56,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:56,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,763 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,763 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,763 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:56,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6807fdd6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:56,764 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:56,764 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,765 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,765 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:56,765 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:56,765 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:56,765 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,765 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:56,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:56,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,766 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:56,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:56,768 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:56,768 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:56,768 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:56,768 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:56,768 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:56,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,769 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,771 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:56,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:56,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:56,772 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,773 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:56,773 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:56,775 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,775 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530276856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530276857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/9b6904ef696947c3bb19593c3b36fe2c 2023-04-03 13:56:56,872 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/5c131959cff6492b975bb82a31ec10be as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5c131959cff6492b975bb82a31ec10be 2023-04-03 13:56:56,873 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:56,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:56,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:56,877 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5c131959cff6492b975bb82a31ec10be, entries=150, sequenceid=66, filesize=25.1 K 2023-04-03 13:56:56,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:56,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:56,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/e8960e1c3f2646819af9999b1831accf as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e8960e1c3f2646819af9999b1831accf 2023-04-03 13:56:56,878 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:56,878 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,878 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:56,878 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:56,880 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,882 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:56,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:56,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:56,883 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:56,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:56,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e8960e1c3f2646819af9999b1831accf, entries=150, sequenceid=66, filesize=11.6 K 2023-04-03 13:56:56,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:56,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:56,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:56,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:56,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:56,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/9b6904ef696947c3bb19593c3b36fe2c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/9b6904ef696947c3bb19593c3b36fe2c 2023-04-03 13:56:56,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,886 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:56,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527b06d4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:56,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:56,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/9b6904ef696947c3bb19593c3b36fe2c, entries=150, sequenceid=66, filesize=11.6 K 2023-04-03 13:56:56,894 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 67b1fdef2f316cba9760895849ff9624 in 901ms, sequenceid=66, compaction requested=true 2023-04-03 13:56:56,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:56,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:56,894 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:56,894 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:56,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:56,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:56,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:56,897 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,897 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:56,897 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:56,898 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:56,901 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:56,901 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:56,901 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:56,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,901 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/B is initiating minor compaction (all files) 2023-04-03 13:56:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,901 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/A is initiating minor compaction (all files) 2023-04-03 13:56:56,902 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/A in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,902 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5d66fc1b311f40b7aaab3da0da8a56a0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f94d62b93f284513b601b2f403466fe5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5c131959cff6492b975bb82a31ec10be] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=82.2 K 2023-04-03 13:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,901 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,902 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/B in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,902 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:56,902 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2515b541ee814bd081832cbdd9fb77d0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fb8ff36826a54dfa9bce78d577aec580, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e8960e1c3f2646819af9999b1831accf] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=34.8 K 2023-04-03 13:56:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,903 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5d66fc1b311f40b7aaab3da0da8a56a0, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680530214714 2023-04-03 13:56:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,903 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2515b541ee814bd081832cbdd9fb77d0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680530214716 2023-04-03 13:56:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,903 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f94d62b93f284513b601b2f403466fe5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680530214752 2023-04-03 13:56:56,903 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fb8ff36826a54dfa9bce78d577aec580, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680530214752 2023-04-03 13:56:56,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,904 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e8960e1c3f2646819af9999b1831accf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680530215380 2023-04-03 13:56:56,904 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5c131959cff6492b975bb82a31ec10be, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680530215380 2023-04-03 13:56:56,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:56,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:56,905 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:56,905 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:56,906 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:56,906 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:56,906 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:56,913 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#B#compaction#102 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:56,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:56,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:56,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,926 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#A#compaction#103 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:56,929 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,929 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,929 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:56,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:56,929 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:56,929 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:56,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:56,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:56,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:56,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:56,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:56,930 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:56,930 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:56,930 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:56,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:56,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:56,944 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/c25aa519d0744f1e94a1497cfb95cb17 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/c25aa519d0744f1e94a1497cfb95cb17 2023-04-03 13:56:56,950 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/B of 67b1fdef2f316cba9760895849ff9624 into c25aa519d0744f1e94a1497cfb95cb17(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:56,951 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:56,951 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/B, priority=13, startTime=1680530216894; duration=0sec 2023-04-03 13:56:56,951 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:56:56,951 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:56:56,952 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:56:56,952 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/C is initiating minor compaction (all files) 2023-04-03 13:56:56,952 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/C in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:56,952 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d9c3bff28b114e94be1258b7214482d2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/65a165921a274cc888a3adaa2d2387a3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/9b6904ef696947c3bb19593c3b36fe2c] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=34.8 K 2023-04-03 13:56:56,953 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d9c3bff28b114e94be1258b7214482d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680530214716 2023-04-03 13:56:56,953 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 65a165921a274cc888a3adaa2d2387a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680530214752 2023-04-03 13:56:56,954 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 9b6904ef696947c3bb19593c3b36fe2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680530215380 2023-04-03 13:56:56,968 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#C#compaction#104 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:56:57,030 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:57,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:57,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:57,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:57,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:57,034 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:57,034 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:57,034 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:57,034 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:57,034 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:57,034 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:57,035 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:57,035 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:57,035 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,035 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:57,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:57,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:57,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:57,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:57,036 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:57,036 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:57,036 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:57,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:57,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:57,037 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:57,037 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:57,037 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 13:56:57,037 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:56:57,037 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:57,037 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:56:57,037 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:57,037 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:56:57,037 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:57,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:57,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:57,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530277134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530277136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530277137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530277240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530277240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530277240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,382 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/c5018dd164084509bbe4ef7f5af95874 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c5018dd164084509bbe4ef7f5af95874 2023-04-03 13:56:57,388 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/A of 67b1fdef2f316cba9760895849ff9624 into c5018dd164084509bbe4ef7f5af95874(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:57,388 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:57,388 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/A, priority=13, startTime=1680530216894; duration=0sec 2023-04-03 13:56:57,388 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/0098bfe86c384317944cf6e984693ee4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0098bfe86c384317944cf6e984693ee4 2023-04-03 13:56:57,388 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:57,394 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/C of 67b1fdef2f316cba9760895849ff9624 into 0098bfe86c384317944cf6e984693ee4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:56:57,394 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:57,394 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/C, priority=13, startTime=1680530216894; duration=0sec 2023-04-03 13:56:57,394 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:56:57,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530277444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530277444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530277444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,460 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403b4497921fe1a467284ed959c7619dc39 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b4497921fe1a467284ed959c7619dc39 2023-04-03 13:56:57,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530277746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530277746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:57,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530277747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:57,890 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/0b8831e9785f44a1922058ba259f7b02 2023-04-03 13:56:57,902 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/b232854552fe41f88a08ce27cf4a1b6d 2023-04-03 13:56:58,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:58,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530278247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:58,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530278252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:58,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530278252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,313 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/0ee12055cf494bf4a04970fcefc108f9 2023-04-03 13:56:58,318 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/0b8831e9785f44a1922058ba259f7b02 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b8831e9785f44a1922058ba259f7b02 2023-04-03 13:56:58,323 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b8831e9785f44a1922058ba259f7b02, entries=150, sequenceid=85, filesize=25.1 K 2023-04-03 13:56:58,324 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/b232854552fe41f88a08ce27cf4a1b6d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/b232854552fe41f88a08ce27cf4a1b6d 2023-04-03 13:56:58,329 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/b232854552fe41f88a08ce27cf4a1b6d, entries=150, sequenceid=85, filesize=11.6 K 2023-04-03 13:56:58,335 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/0ee12055cf494bf4a04970fcefc108f9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0ee12055cf494bf4a04970fcefc108f9 2023-04-03 13:56:58,349 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0ee12055cf494bf4a04970fcefc108f9, entries=150, sequenceid=85, filesize=11.6 K 2023-04-03 13:56:58,349 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 67b1fdef2f316cba9760895849ff9624 in 1312ms, sequenceid=85, compaction requested=false 2023-04-03 13:56:58,350 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:58,350 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:58,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:58,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:58,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:58,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:58,351 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:58,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:58,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:58,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:58,353 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:58,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d75975[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:58,353 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:58,353 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:58,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:58,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:58,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:58,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:58,354 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,354 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:58,354 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:58,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:58,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:58,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:58,359 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:58,359 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:58,359 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:58,359 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:58,359 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:58,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:58,365 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,365 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,365 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:58,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:58,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:58,366 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:58,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:58,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:58,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:58,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,368 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:58,368 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:58,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:58,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:58,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:58,468 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:56:58,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:56:58,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:56:58,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:56:58,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:56:58,484 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:56:58,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:58,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:56:58,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:56:58,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:58,491 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:58,491 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:58,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:56:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:58,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:56:58,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:58,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:56:58,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:56:58,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:56:58,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:56:58,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:56:58,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:56:58,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:56:58,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:56:58,518 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:58,518 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:56:58,518 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 13:56:58,524 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:56:58,524 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:58,524 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:56:58,524 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:58,524 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:56:58,524 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:56:58,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:56:58,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:56:58,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:58,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530278906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:58,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:58,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530278909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530279011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530279015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,020 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040372aeb99241c641498cac29b77164689a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040372aeb99241c641498cac29b77164689a 2023-04-03 13:56:59,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530279215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530279219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530279256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530279260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530279266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,435 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=101, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/bb20c68c6f72420b82b9522382450204 2023-04-03 13:56:59,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530279520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:56:59,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530279522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,850 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/e2b688949b22459496a3cd0d9a0558ab 2023-04-03 13:56:59,879 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=101 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/3140743c25fd4ecaa1725272c116748e 2023-04-03 13:56:59,884 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/bb20c68c6f72420b82b9522382450204 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bb20c68c6f72420b82b9522382450204 2023-04-03 13:56:59,889 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bb20c68c6f72420b82b9522382450204, entries=150, sequenceid=101, filesize=25.1 K 2023-04-03 13:56:59,890 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/e2b688949b22459496a3cd0d9a0558ab as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e2b688949b22459496a3cd0d9a0558ab 2023-04-03 13:56:59,895 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e2b688949b22459496a3cd0d9a0558ab, entries=150, sequenceid=101, filesize=11.6 K 2023-04-03 13:56:59,911 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/3140743c25fd4ecaa1725272c116748e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3140743c25fd4ecaa1725272c116748e 2023-04-03 13:56:59,921 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3140743c25fd4ecaa1725272c116748e, entries=150, sequenceid=101, filesize=11.6 K 2023-04-03 13:56:59,922 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 67b1fdef2f316cba9760895849ff9624 in 1404ms, sequenceid=101, compaction requested=true 2023-04-03 13:56:59,922 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:56:59,922 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:56:59,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:56:59,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:56:59,923 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,926 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:56:59,926 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:56:59,927 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:59,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:59,928 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:59,928 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:56:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:59,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:56:59,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8e9c0cf[Count = 0] remaining members to acquire global barrier 2023-04-03 13:56:59,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:56:59,932 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:59,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:59,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:59,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:56:59,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:56:59,934 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,934 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:56:59,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:56:59,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:56:59,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:59,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:56:59,936 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:59,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:56:59,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:56:59,939 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:56:59,939 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:56:59,939 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:56:59,939 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:56:59,940 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:56:59,949 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:59,949 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:59,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:59,949 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:56:59,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:56:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:56:59,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:56:59,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:59,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,952 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:59,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:56:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:56:59,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,957 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,957 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:59,957 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:56:59,957 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:56:59,957 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:56:59,957 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:56:59,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:56:59,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:56:59,957 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:59,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:56:59,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:56:59,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:56:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:56:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:56:59,957 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:56:59,958 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:56:59,958 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:56:59,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:56:59,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:56:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:00,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 13:57:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:00,026 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:00,027 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:00,027 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:00,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:00,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:00,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530280043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530280043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,055 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403dffd4392091b4b7ab8a9a499b1b94dd8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403dffd4392091b4b7ab8a9a499b1b94dd8 2023-04-03 13:57:00,058 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:00,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:00,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:00,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:00,078 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:00,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:00,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:00,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:00,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:00,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:00,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:00,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:00,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:00,090 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:00,090 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:00,090 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:00,090 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:00,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:00,092 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,092 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,092 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:00,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd76549[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:00,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:00,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,098 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:00,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:00,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:00,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:00,099 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:00,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:00,102 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:00,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:00,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:00,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:00,102 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:00,103 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,103 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,104 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,103 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,108 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:00,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:00,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,109 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:00,109 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:00,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:00,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:00,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530280151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:00,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530280154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,209 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:00,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:00,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:00,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:00,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:00,213 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:00,213 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,213 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:00,213 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:00,213 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,213 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,214 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:00,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:00,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:00,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:00,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:00,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:00,216 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,216 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:00,216 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:00,217 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,217 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:00,217 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:00,217 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,217 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:00,217 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:00,218 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:00,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@551095c1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:00,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:00,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,220 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,220 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:00,220 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:00,220 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:00,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,221 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:00,221 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:00,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:00,225 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:00,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:00,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:00,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:00,226 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:00,234 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,234 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,235 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,235 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,238 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:00,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:00,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,241 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:00,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,241 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:00,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,245 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,245 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:00,341 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:00,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:00,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:00,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:00,346 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:00,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:00,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:00,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,348 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,348 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:00,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:00,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:00,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:00,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:00,352 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,352 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:00,352 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:00,352 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:00,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:00,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530280354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,357 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,357 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,358 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,358 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530280358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:00,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c1572e7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:00,361 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:00,361 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,368 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:00,368 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:00,368 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:00,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:00,374 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,374 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:00,374 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,374 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:00,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:00,379 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:00,379 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:00,379 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:00,379 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:00,379 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:00,384 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,384 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,384 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,385 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,386 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,390 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:00,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:00,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:00,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,391 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:00,391 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:00,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,488 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=120, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/10b6e7cb7d5a41e69a2a530ac92a5353 2023-04-03 13:57:00,491 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:00,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:00,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:00,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:00,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:00,495 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:00,495 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,495 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:00,495 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:00,495 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,495 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,496 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:00,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:00,498 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,498 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:00,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:00,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:00,502 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:00,502 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,502 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:00,502 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:00,502 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:00,502 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,502 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:00,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:00,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:00,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:00,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:00,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:00,507 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,507 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:00,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:00,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@abc2c12[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:00,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,509 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:00,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:00,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:00,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:00,511 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:00,511 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,511 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:00,511 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:00,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:00,513 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:00,513 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:00,513 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:00,513 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:00,513 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:00,514 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,514 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,514 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:00,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:00,518 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,518 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:00,518 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:00,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,619 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:00,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:00,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:00,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:00,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:00,622 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:00,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:00,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:00,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,623 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:00,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:00,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:00,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:00,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:00,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:00,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:00,624 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,624 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:00,625 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:00,625 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:00,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:00,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:00,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:00,626 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:00,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:00,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e25fd21[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:00,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,628 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:00,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:00,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:00,628 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,628 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:00,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:00,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:00,631 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:00,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:00,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:00,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:00,631 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:00,631 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,631 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,632 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,633 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,635 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,635 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:00,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:00,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,635 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:00,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:00,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530280659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:00,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530280663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,736 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:00,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:00,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:00,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:00,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:00,738 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:00,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:00,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:00,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,740 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:00,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:00,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:00,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:00,741 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,741 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:00,741 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:00,741 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:00,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:00,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:00,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:00,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f4ad3b9[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:00,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:00,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:00,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:00,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:00,746 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,746 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:00,747 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,747 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:00,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:00,748 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:00,748 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:00,748 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:00,748 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:00,748 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:00,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:00,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:00,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,753 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:00,753 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,753 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,753 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:00,853 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:00,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:00,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:00,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:00,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:00,856 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:00,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:00,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:00,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,857 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,857 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,857 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,857 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:00,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:00,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:00,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:00,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:00,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:00,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:00,858 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,858 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:00,858 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:00,858 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:00,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:00,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:00,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:00,859 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,860 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,860 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:00,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:00,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@582f0cfd[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:00,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,862 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,862 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:00,862 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,862 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:00,862 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:00,862 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:00,862 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,863 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,863 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,863 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:00,864 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:00,865 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:00,865 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:00,865 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:00,865 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:00,865 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,865 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,867 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:00,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,869 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:00,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:00,870 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:00,870 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:00,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/543434ddd407431b80306c3c513a68be 2023-04-03 13:57:00,970 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:00,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:00,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:00,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:00,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:00,973 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:00,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:00,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:00,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,974 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:00,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:00,976 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,976 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:00,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:00,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:00,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:00,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:00,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:00,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:00,986 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:00,986 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:00,986 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,986 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:00,986 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:00,986 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:00,986 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:00,986 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:00,986 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,986 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:00,986 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:00,992 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,992 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,993 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:00,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:00,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:00,995 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:00,995 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33186756[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:00,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:00,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:00,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:00,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:00,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:00,999 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:00,999 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:00,999 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:00,999 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:00,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:01,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:01,002 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:01,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:01,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:01,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:01,002 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:01,003 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,003 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:01,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,004 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:01,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,007 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,009 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:01,009 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:01,009 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,009 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,009 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:01,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:01,009 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,009 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:01,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:01,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:01,010 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:01,010 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:01,110 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:01,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:01,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:01,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:01,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:01,113 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:01,113 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:01,113 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:01,113 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:01,113 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,113 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:01,114 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:01,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:01,114 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,115 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:01,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:01,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:01,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:01,115 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:01,115 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:01,115 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:01,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:01,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:01,116 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:01,116 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:01,116 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:01,116 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:01,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:01,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:01,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:01,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:01,117 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:01,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:01,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:01,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@729548e1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:01,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,119 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:01,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:01,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:01,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:01,120 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:01,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:01,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:01,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:01,122 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:01,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:01,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:01,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:01,122 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:01,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:01,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,125 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,126 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:01,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:01,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:01,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:01,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:01,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:01,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:01,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,129 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:01,129 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:01,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:01,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:01,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:01,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530281162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:01,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530281169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,229 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:01,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:01,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:01,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:01,231 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:01,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:01,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:01,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:01,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:01,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:01,233 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,233 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:01,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:01,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:01,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:01,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:01,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:01,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:01,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:01,234 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:01,234 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:01,235 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:01,235 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:01,235 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:01,235 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:01,235 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,235 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:01,235 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:01,236 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,236 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:01,236 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,236 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:01,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:01,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bab9a11[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:01,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,238 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,238 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:01,238 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:01,238 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:01,238 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,238 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:01,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:01,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:01,239 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:01,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:01,241 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:01,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:01,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:01,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:01,241 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:01,242 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,242 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:01,242 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:01,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:01,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:01,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,244 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:01,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:01,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:01,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,246 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:01,246 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:01,246 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:01,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:01,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:01,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:01,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:01,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:01,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:01,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530281262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,264 DEBUG [Thread-779] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4127 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:01,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:01,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530281274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,276 DEBUG [Thread-781] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:01,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:01,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530281275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,282 DEBUG [Thread-785] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:01,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/35b0d88c1fe14c42b471f1ef6d684376 2023-04-03 13:57:01,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/10b6e7cb7d5a41e69a2a530ac92a5353 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10b6e7cb7d5a41e69a2a530ac92a5353 2023-04-03 13:57:01,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10b6e7cb7d5a41e69a2a530ac92a5353, entries=200, sequenceid=120, filesize=31.9 K 2023-04-03 13:57:01,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/543434ddd407431b80306c3c513a68be as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/543434ddd407431b80306c3c513a68be 2023-04-03 13:57:01,332 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/543434ddd407431b80306c3c513a68be, entries=150, sequenceid=120, filesize=11.6 K 2023-04-03 13:57:01,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/35b0d88c1fe14c42b471f1ef6d684376 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/35b0d88c1fe14c42b471f1ef6d684376 2023-04-03 13:57:01,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/35b0d88c1fe14c42b471f1ef6d684376, entries=150, sequenceid=120, filesize=11.6 K 2023-04-03 13:57:01,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 67b1fdef2f316cba9760895849ff9624 in 1312ms, sequenceid=120, compaction requested=true 2023-04-03 13:57:01,338 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:01,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:01,338 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:01,338 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:01,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:01,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:01,339 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110027 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:01,339 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:01,339 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/A is initiating minor compaction (all files) 2023-04-03 13:57:01,339 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/B is initiating minor compaction (all files) 2023-04-03 13:57:01,340 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/A in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:01,340 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/B in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:01,340 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c5018dd164084509bbe4ef7f5af95874, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b8831e9785f44a1922058ba259f7b02, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bb20c68c6f72420b82b9522382450204, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10b6e7cb7d5a41e69a2a530ac92a5353] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=107.4 K 2023-04-03 13:57:01,340 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/c25aa519d0744f1e94a1497cfb95cb17, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/b232854552fe41f88a08ce27cf4a1b6d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e2b688949b22459496a3cd0d9a0558ab, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/543434ddd407431b80306c3c513a68be] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=46.5 K 2023-04-03 13:57:01,340 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting c5018dd164084509bbe4ef7f5af95874, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680530215380 2023-04-03 13:57:01,340 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c25aa519d0744f1e94a1497cfb95cb17, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680530215380 2023-04-03 13:57:01,340 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 0b8831e9785f44a1922058ba259f7b02, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680530216005 2023-04-03 13:57:01,340 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting b232854552fe41f88a08ce27cf4a1b6d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680530216005 2023-04-03 13:57:01,341 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting bb20c68c6f72420b82b9522382450204, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680530217133 2023-04-03 13:57:01,341 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e2b688949b22459496a3cd0d9a0558ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680530217133 2023-04-03 13:57:01,341 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 10b6e7cb7d5a41e69a2a530ac92a5353, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680530218898 2023-04-03 13:57:01,341 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 543434ddd407431b80306c3c513a68be, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680530218898 2023-04-03 13:57:01,346 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:01,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:01,353 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#B#compaction#114 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:01,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:01,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:01,363 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:01,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:01,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:01,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:01,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:01,364 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:01,364 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:01,364 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:01,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:01,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:01,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:01,368 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/5be6acae10304ffd813b85b8b3729afe as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5be6acae10304ffd813b85b8b3729afe 2023-04-03 13:57:01,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:01,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:01,386 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:01,386 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:01,386 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:01,389 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/B of 67b1fdef2f316cba9760895849ff9624 into 5be6acae10304ffd813b85b8b3729afe(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:01,389 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:01,389 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/B, priority=12, startTime=1680530221338; duration=0sec 2023-04-03 13:57:01,389 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:01,389 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:01,391 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:01,391 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/C is initiating minor compaction (all files) 2023-04-03 13:57:01,391 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/C in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:01,391 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0098bfe86c384317944cf6e984693ee4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0ee12055cf494bf4a04970fcefc108f9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3140743c25fd4ecaa1725272c116748e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/35b0d88c1fe14c42b471f1ef6d684376] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=46.5 K 2023-04-03 13:57:01,392 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 0098bfe86c384317944cf6e984693ee4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680530215380 2023-04-03 13:57:01,392 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 0ee12055cf494bf4a04970fcefc108f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680530216005 2023-04-03 13:57:01,393 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:01,393 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:01,393 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:01,393 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:01,393 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 13:57:01,393 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:01,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:01,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:01,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:01,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:01,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:01,394 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#A#compaction#115 average throughput is 0.62 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:01,395 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 3140743c25fd4ecaa1725272c116748e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=101, earliestPutTs=1680530217133 2023-04-03 13:57:01,396 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 35b0d88c1fe14c42b471f1ef6d684376, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680530218898 2023-04-03 13:57:01,416 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#C#compaction#116 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:01,447 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403b19be94642fa41e7b8b2ba9b31a88b73 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b19be94642fa41e7b8b2ba9b31a88b73 2023-04-03 13:57:01,849 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/e6cbcea0d1344bea95b2c29102f60059 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e6cbcea0d1344bea95b2c29102f60059 2023-04-03 13:57:01,854 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/C of 67b1fdef2f316cba9760895849ff9624 into e6cbcea0d1344bea95b2c29102f60059(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:01,854 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:01,854 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/C, priority=12, startTime=1680530221338; duration=0sec 2023-04-03 13:57:01,854 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:01,857 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/aaeff3addc794d9ead3e6878748f771c 2023-04-03 13:57:01,873 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/a89b82e7a4634914b3bc4b71439d379d 2023-04-03 13:57:02,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:02,172 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:02,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530282193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530282196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,230 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/0b0c898eb9e44404a0229bd7cfeb22cc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b0c898eb9e44404a0229bd7cfeb22cc 2023-04-03 13:57:02,235 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/A of 67b1fdef2f316cba9760895849ff9624 into 0b0c898eb9e44404a0229bd7cfeb22cc(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:02,235 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:02,235 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/A, priority=12, startTime=1680530221338; duration=0sec 2023-04-03 13:57:02,235 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:02,293 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/092b2376d3e047d1af3bef0e9b5fd567 2023-04-03 13:57:02,298 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/aaeff3addc794d9ead3e6878748f771c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/aaeff3addc794d9ead3e6878748f771c 2023-04-03 13:57:02,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530282298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530282298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,302 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/aaeff3addc794d9ead3e6878748f771c, entries=150, sequenceid=134, filesize=25.3 K 2023-04-03 13:57:02,303 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/a89b82e7a4634914b3bc4b71439d379d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a89b82e7a4634914b3bc4b71439d379d 2023-04-03 13:57:02,307 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a89b82e7a4634914b3bc4b71439d379d, entries=150, sequenceid=134, filesize=11.8 K 2023-04-03 13:57:02,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/092b2376d3e047d1af3bef0e9b5fd567 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/092b2376d3e047d1af3bef0e9b5fd567 2023-04-03 13:57:02,314 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/092b2376d3e047d1af3bef0e9b5fd567, entries=150, sequenceid=134, filesize=11.8 K 2023-04-03 13:57:02,315 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 67b1fdef2f316cba9760895849ff9624 in 922ms, sequenceid=134, compaction requested=false 2023-04-03 13:57:02,315 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:02,315 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:02,315 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:02,315 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:02,315 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:02,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:02,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:02,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:02,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:02,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:02,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:02,326 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:02,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc3357d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:02,326 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:02,336 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:02,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:02,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:02,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:02,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:02,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:02,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:02,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:02,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:02,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:02,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:02,340 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:02,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:02,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:02,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:02,340 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:02,341 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:02,341 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,341 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:02,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:02,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:02,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,343 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:02,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:02,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:02,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:02,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:02,345 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:02,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:02,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:02,345 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:02,345 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:02,445 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:02,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:02,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:02,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:02,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:02,449 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:02,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:02,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:02,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:02,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:02,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,450 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:02,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:02,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:02,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:02,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:02,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:02,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:02,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:02,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:02,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:02,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:02,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:02,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:02,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:02,451 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 13:57:02,452 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:02,452 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:02,452 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:02,452 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:02,452 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:02,452 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:02,467 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304031be3d5fce39a4ffea7bfa0762a298a74 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031be3d5fce39a4ffea7bfa0762a298a74 2023-04-03 13:57:02,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:02,505 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:02,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530282516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530282516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530282619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530282619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:02,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530282821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530282822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:02,878 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/9fd4450872274bc888e1a057021dad36 2023-04-03 13:57:03,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:03,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530283124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:03,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530283125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,289 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/609f5760c3764614abdb1ff97ce344d7 2023-04-03 13:57:03,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:03,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:03,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530283629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530283629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,699 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/d78ac82d81be49bcb3c66c2b7ce6a70b 2023-04-03 13:57:03,703 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/9fd4450872274bc888e1a057021dad36 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9fd4450872274bc888e1a057021dad36 2023-04-03 13:57:03,707 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9fd4450872274bc888e1a057021dad36, entries=150, sequenceid=155, filesize=25.3 K 2023-04-03 13:57:03,708 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/609f5760c3764614abdb1ff97ce344d7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/609f5760c3764614abdb1ff97ce344d7 2023-04-03 13:57:03,712 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/609f5760c3764614abdb1ff97ce344d7, entries=150, sequenceid=155, filesize=11.8 K 2023-04-03 13:57:03,713 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/d78ac82d81be49bcb3c66c2b7ce6a70b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d78ac82d81be49bcb3c66c2b7ce6a70b 2023-04-03 13:57:03,717 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d78ac82d81be49bcb3c66c2b7ce6a70b, entries=150, sequenceid=155, filesize=11.8 K 2023-04-03 13:57:03,718 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 67b1fdef2f316cba9760895849ff9624 in 1267ms, sequenceid=155, compaction requested=true 2023-04-03 13:57:03,718 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:03,718 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:03,718 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:03,718 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:03,718 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,718 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:03,718 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:03,719 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,719 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:03,720 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:03,720 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:03,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:03,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b91c784[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:03,721 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:03,721 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:03,722 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:03,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:03,722 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,722 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:03,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:03,723 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:03,723 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:03,723 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:03,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:03,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:03,726 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:03,726 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:03,726 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:03,726 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:03,726 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:03,727 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,727 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:03,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:03,727 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:03,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,728 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:03,729 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,730 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,730 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:03,730 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:03,730 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:03,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:03,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:03,730 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:03,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:03,730 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:03,731 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,731 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:03,731 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:03,731 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:03,731 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:03,731 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:03,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:03,831 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:03,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:03,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:03,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:03,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:03,833 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:03,834 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:03,834 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:03,834 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:03,834 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,834 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:03,835 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:03,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:03,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:03,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:03,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:03,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:03,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:03,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:03,836 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:03,836 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:03,836 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:03,838 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:03,839 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:03,839 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 13:57:03,839 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:03,839 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:03,839 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:03,839 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:03,839 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:03,839 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:04,257 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304034f72fd27c2a64cb5bdc245620f247989 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034f72fd27c2a64cb5bdc245620f247989 2023-04-03 13:57:04,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:04,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:04,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:04,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530284657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:04,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:04,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530284658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:04,662 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=168, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/9c8a9a47d6b04bb0b0bac5c72bc8cde1 2023-04-03 13:57:04,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:04,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530284760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:04,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:04,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530284761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:04,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:04,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530284963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:04,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:04,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530284963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,082 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/fa6fcce71d494a5a80c8a78caa829b29 2023-04-03 13:57:05,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530285267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530285269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530285272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,273 DEBUG [Thread-779] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:05,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530285311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,313 DEBUG [Thread-781] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:05,324 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530285322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,326 DEBUG [Thread-785] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8188 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:05,492 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/3f1f4e1449074220843ee2f0e6c1c2aa 2023-04-03 13:57:05,496 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/9c8a9a47d6b04bb0b0bac5c72bc8cde1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9c8a9a47d6b04bb0b0bac5c72bc8cde1 2023-04-03 13:57:05,502 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9c8a9a47d6b04bb0b0bac5c72bc8cde1, entries=150, sequenceid=168, filesize=25.3 K 2023-04-03 13:57:05,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/fa6fcce71d494a5a80c8a78caa829b29 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fa6fcce71d494a5a80c8a78caa829b29 2023-04-03 13:57:05,507 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fa6fcce71d494a5a80c8a78caa829b29, entries=150, sequenceid=168, filesize=11.8 K 2023-04-03 13:57:05,508 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/3f1f4e1449074220843ee2f0e6c1c2aa as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3f1f4e1449074220843ee2f0e6c1c2aa 2023-04-03 13:57:05,515 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3f1f4e1449074220843ee2f0e6c1c2aa, entries=150, sequenceid=168, filesize=11.8 K 2023-04-03 13:57:05,516 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 67b1fdef2f316cba9760895849ff9624 in 1677ms, sequenceid=168, compaction requested=true 2023-04-03 13:57:05,516 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:05,516 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:05,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:05,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:05,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:05,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:05,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:05,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:05,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:05,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:05,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:05,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:05,520 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:05,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@131b5c57[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:05,520 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:05,520 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:05,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:05,521 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:05,521 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:05,521 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:05,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:05,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,521 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:05,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:05,521 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:05,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:05,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:05,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:05,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:05,525 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:05,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:05,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:05,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:05,525 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:05,529 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,529 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,529 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:05,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:05,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:05,530 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:05,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:05,531 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:05,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:05,532 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:05,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:05,532 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:05,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:05,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:05,532 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:05,533 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:05,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:05,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:05,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:05,633 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:05,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:05,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:05,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:05,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:05,636 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:05,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:05,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:05,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:05,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:05,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,638 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:05,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:05,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:05,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:05,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:05,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:05,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:05,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:05,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:05,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:05,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:05,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:05,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:05,639 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:05,639 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:05,639 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 13:57:05,639 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:05,640 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:05,640 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:05,640 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:05,640 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:05,640 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:05,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:05,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:05,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530285788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530285791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530285892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:05,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:05,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530285895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,052 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403100b57641480446982885197c233643e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403100b57641480446982885197c233643e 2023-04-03 13:57:06,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:06,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530286094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:06,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530286099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:06,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530286398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:06,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530286404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,457 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=187, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/c40b2d984b03427aa35fe5948feceeb3 2023-04-03 13:57:06,867 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/0890820a2bba44a2b9c80aeba414775c 2023-04-03 13:57:06,881 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/a3092c9fc13c4b2892d978d59f09ffdd 2023-04-03 13:57:06,885 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/c40b2d984b03427aa35fe5948feceeb3 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c40b2d984b03427aa35fe5948feceeb3 2023-04-03 13:57:06,889 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c40b2d984b03427aa35fe5948feceeb3, entries=150, sequenceid=187, filesize=25.3 K 2023-04-03 13:57:06,890 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/0890820a2bba44a2b9c80aeba414775c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/0890820a2bba44a2b9c80aeba414775c 2023-04-03 13:57:06,894 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/0890820a2bba44a2b9c80aeba414775c, entries=150, sequenceid=187, filesize=11.8 K 2023-04-03 13:57:06,895 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/a3092c9fc13c4b2892d978d59f09ffdd as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/a3092c9fc13c4b2892d978d59f09ffdd 2023-04-03 13:57:06,898 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/a3092c9fc13c4b2892d978d59f09ffdd, entries=150, sequenceid=187, filesize=11.8 K 2023-04-03 13:57:06,899 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 67b1fdef2f316cba9760895849ff9624 in 1260ms, sequenceid=187, compaction requested=true 2023-04-03 13:57:06,899 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:06,899 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:06,899 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:06,899 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:06,899 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,899 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:06,899 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:06,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:06,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 13:57:06,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:06,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:06,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:06,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:06,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:06,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:06,904 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,904 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:06,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:06,904 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:06,905 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:06,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:06,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54273737[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:06,906 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:06,906 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:06,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:06,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:06,908 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:06,908 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:06,908 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:06,908 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,908 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:06,909 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:06,909 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:06,909 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,909 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:06,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:06,912 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:06,912 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:06,912 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:06,912 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:06,912 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:06,913 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:06,913 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:06,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:06,913 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:06,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:06,913 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,914 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:06,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:06,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:06,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:06,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:06,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:06,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:06,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:06,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:06,916 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:06,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,916 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:06,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:06,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:06,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:06,917 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:06,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:06,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530286928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:06,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:06,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530286929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,016 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,019 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,020 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,020 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,020 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,020 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,021 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,021 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,022 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,022 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,022 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,022 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,023 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,024 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766d8813[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,024 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,025 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,025 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,025 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,025 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,026 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,027 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,027 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,027 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,027 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,027 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,028 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,028 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,028 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,028 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,031 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,031 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:07,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,031 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,031 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,032 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:07,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530287033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:07,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530287033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,132 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,134 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,134 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,135 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,135 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,135 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,135 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,135 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,135 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,136 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,136 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,137 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,137 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,137 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,137 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,138 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,138 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,138 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,138 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3305ccbd[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,140 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,140 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,140 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,140 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,140 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,141 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,141 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,143 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,143 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,143 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,143 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,143 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,145 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,145 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,147 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,147 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,148 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,148 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,148 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:07,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530287235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:07,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530287235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,248 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,251 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,251 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,251 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,251 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,251 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,251 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,252 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,252 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,252 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,252 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,254 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,254 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,254 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,254 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,255 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,255 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,256 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,256 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a35c76e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,257 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,257 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,257 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,257 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,258 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,258 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,258 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,260 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,260 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,260 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,260 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,260 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,261 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,261 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,261 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,262 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,262 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,264 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:07,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,264 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,264 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,265 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,315 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304037b2fd8d65f9c4b348c2a0fd37fbcf296 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037b2fd8d65f9c4b348c2a0fd37fbcf296 2023-04-03 13:57:07,365 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,367 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,369 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,369 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,370 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,370 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,370 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,370 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,371 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,371 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,371 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,371 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c6d990[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,372 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,373 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,373 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,373 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,374 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,375 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,375 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,375 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,375 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,375 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,376 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,377 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,379 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,379 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,379 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:07,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,479 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,482 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,482 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,482 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,482 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,483 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,483 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,483 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,485 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,485 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,485 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,485 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,485 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,486 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,486 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,487 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,487 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@df1e66b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,488 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,488 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,488 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,488 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,488 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,489 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,489 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,489 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,489 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,489 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,489 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,491 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,491 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,491 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,491 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,491 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,493 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,494 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,495 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,495 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,495 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,495 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,495 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,495 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,495 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,495 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,495 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,495 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,495 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:07,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41082 deadline: 1680530287541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:07,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530287542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,595 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,598 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,599 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,599 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,599 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,599 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,601 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,601 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,601 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,601 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,601 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,612 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,612 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,613 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,613 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,614 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@100ca34c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,614 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,619 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,619 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,619 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,623 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,623 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,623 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,623 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,624 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,624 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,636 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,636 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:07,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,720 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/8993ff439e144d6388811bacfd87cfbe 2023-04-03 13:57:07,736 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,739 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,740 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,741 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,741 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,742 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,742 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,742 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,742 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44d60894[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,746 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,746 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,747 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,747 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,748 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,748 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,748 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,749 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,752 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,752 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,852 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,855 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,858 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65fd53c7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,860 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,861 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,862 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,862 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,863 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,863 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,864 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,864 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,864 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,865 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,867 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:07,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,868 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,868 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,968 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:07,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:07,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:07,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:07,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:07,971 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:07,971 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,971 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:07,971 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:07,971 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,971 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:07,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:07,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:07,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:07,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:07,973 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:07,973 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:07,973 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:07,973 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:07,973 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:07,973 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,973 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:07,974 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:07,974 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:07,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:07,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,975 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:07,976 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:07,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67bd23ad[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:07,976 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:07,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:07,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:07,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:07,977 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,978 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:07,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:07,981 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:07,981 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:07,981 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:07,981 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:07,981 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:07,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:07,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:07,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:07,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,984 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:07,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:07,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:07,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:07,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:07,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:07,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:07,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:07,997 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:07,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:07,997 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:07,997 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,997 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:07,997 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:07,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:07,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:08,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530288044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:08,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530288046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,097 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:08,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:08,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:08,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:08,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:08,100 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:08,100 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,100 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:08,100 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:08,100 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,100 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,101 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:08,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:08,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:08,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:08,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:08,102 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:08,102 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,102 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:08,102 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:08,102 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:08,102 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,102 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:08,103 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:08,103 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,103 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:08,103 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:08,103 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,103 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:08,103 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:08,104 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:08,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125eead5[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:08,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:08,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,106 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:08,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:08,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:08,106 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,106 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:08,106 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:08,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:08,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:08,108 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:08,108 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:08,108 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:08,108 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:08,108 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:08,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,109 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:08,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,112 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,112 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:08,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:08,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:08,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:08,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:08,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:08,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,113 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:08,113 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:08,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/ce6fb5b571d941ffbd638e56c774b2a3 2023-04-03 13:57:08,213 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:08,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:08,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:08,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:08,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:08,215 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:08,216 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,216 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:08,216 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:08,216 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,216 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,217 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,217 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,217 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,217 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:08,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:08,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:08,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:08,218 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,218 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:08,218 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:08,218 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:08,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:08,219 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,219 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,219 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,219 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:08,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6638d2a2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:08,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:08,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,221 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:08,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:08,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:08,221 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,221 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:08,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:08,221 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:08,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:08,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:08,223 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:08,223 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:08,223 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:08,223 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:08,223 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:08,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,226 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,228 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,228 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:08,228 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:08,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:08,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:08,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:08,228 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:08,228 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:08,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:08,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,329 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:08,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:08,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:08,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:08,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:08,331 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:08,332 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,332 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:08,332 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:08,332 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,332 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,333 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,333 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,333 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,333 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:08,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:08,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:08,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:08,334 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,334 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:08,334 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:08,334 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:08,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:08,335 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:08,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:08,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69d4c6c9[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:08,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:08,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:08,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:08,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:08,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:08,338 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:08,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:08,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:08,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:08,339 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:08,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:08,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:08,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:08,340 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:08,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,341 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:08,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:08,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:08,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:08,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,344 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:08,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:08,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,344 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:08,445 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:08,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:08,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:08,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:08,448 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:08,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:08,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:08,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,450 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:08,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:08,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:08,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:08,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:08,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:08,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:08,452 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,452 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,452 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:08,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@677a2015[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:08,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:08,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:08,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:08,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:08,477 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:08,477 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,477 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:08,478 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:08,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:08,480 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:08,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:08,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:08,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:08,480 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:08,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:08,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,488 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:08,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:08,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:08,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:08,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:08,496 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:08,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,496 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:08,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,496 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:08,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:08,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:08,497 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:08,497 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,541 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/e3c6e59cb2094bf0b62e9825765d36c4 2023-04-03 13:57:08,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/8993ff439e144d6388811bacfd87cfbe as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/8993ff439e144d6388811bacfd87cfbe 2023-04-03 13:57:08,549 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/8993ff439e144d6388811bacfd87cfbe, entries=200, sequenceid=201, filesize=32.1 K 2023-04-03 13:57:08,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/ce6fb5b571d941ffbd638e56c774b2a3 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/ce6fb5b571d941ffbd638e56c774b2a3 2023-04-03 13:57:08,554 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/ce6fb5b571d941ffbd638e56c774b2a3, entries=150, sequenceid=201, filesize=11.8 K 2023-04-03 13:57:08,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/e3c6e59cb2094bf0b62e9825765d36c4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e3c6e59cb2094bf0b62e9825765d36c4 2023-04-03 13:57:08,560 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e3c6e59cb2094bf0b62e9825765d36c4, entries=150, sequenceid=201, filesize=11.8 K 2023-04-03 13:57:08,561 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 67b1fdef2f316cba9760895849ff9624 in 1660ms, sequenceid=201, compaction requested=true 2023-04-03 13:57:08,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:08,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:08,561 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:57:08,561 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:57:08,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:08,561 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:08,563 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72306 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:57:08,563 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162456 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:57:08,563 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/B is initiating minor compaction (all files) 2023-04-03 13:57:08,563 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/A is initiating minor compaction (all files) 2023-04-03 13:57:08,563 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/B in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,563 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/A in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,563 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5be6acae10304ffd813b85b8b3729afe, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a89b82e7a4634914b3bc4b71439d379d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/609f5760c3764614abdb1ff97ce344d7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fa6fcce71d494a5a80c8a78caa829b29, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/0890820a2bba44a2b9c80aeba414775c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/ce6fb5b571d941ffbd638e56c774b2a3] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=70.6 K 2023-04-03 13:57:08,563 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b0c898eb9e44404a0229bd7cfeb22cc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/aaeff3addc794d9ead3e6878748f771c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9fd4450872274bc888e1a057021dad36, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9c8a9a47d6b04bb0b0bac5c72bc8cde1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c40b2d984b03427aa35fe5948feceeb3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/8993ff439e144d6388811bacfd87cfbe] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=158.6 K 2023-04-03 13:57:08,564 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5be6acae10304ffd813b85b8b3729afe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680530218898 2023-04-03 13:57:08,564 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 0b0c898eb9e44404a0229bd7cfeb22cc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680530218898 2023-04-03 13:57:08,564 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting a89b82e7a4634914b3bc4b71439d379d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530220038 2023-04-03 13:57:08,564 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting aaeff3addc794d9ead3e6878748f771c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530220038 2023-04-03 13:57:08,564 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 609f5760c3764614abdb1ff97ce344d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680530222191 2023-04-03 13:57:08,564 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9fd4450872274bc888e1a057021dad36, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680530222191 2023-04-03 13:57:08,565 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9c8a9a47d6b04bb0b0bac5c72bc8cde1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680530222514 2023-04-03 13:57:08,565 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting c40b2d984b03427aa35fe5948feceeb3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680530224650 2023-04-03 13:57:08,565 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fa6fcce71d494a5a80c8a78caa829b29, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680530222514 2023-04-03 13:57:08,565 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 8993ff439e144d6388811bacfd87cfbe, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680530225780 2023-04-03 13:57:08,566 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 0890820a2bba44a2b9c80aeba414775c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680530224650 2023-04-03 13:57:08,566 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ce6fb5b571d941ffbd638e56c774b2a3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680530225780 2023-04-03 13:57:08,588 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#B#compaction#133 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:08,591 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#A#compaction#132 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:08,597 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:08,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:08,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:08,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:08,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:08,602 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:08,602 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,602 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:08,602 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:08,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,603 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:08,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:08,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:08,604 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:08,604 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:08,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:08,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:08,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:08,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:08,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:08,606 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:08,606 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:08,606 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:08,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:08,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:08,610 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:08,610 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:08,610 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 13:57:08,611 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:08,611 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:08,611 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:08,611 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:08,611 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:08,611 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:09,012 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/1ff076284c444b4ba994b95b44293283 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1ff076284c444b4ba994b95b44293283 2023-04-03 13:57:09,016 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/B of 67b1fdef2f316cba9760895849ff9624 into 1ff076284c444b4ba994b95b44293283(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:09,016 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:09,016 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/B, priority=10, startTime=1680530228561; duration=0sec 2023-04-03 13:57:09,016 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:09,016 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:57:09,018 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72306 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:57:09,018 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/C is initiating minor compaction (all files) 2023-04-03 13:57:09,018 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/C in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,018 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e6cbcea0d1344bea95b2c29102f60059, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/092b2376d3e047d1af3bef0e9b5fd567, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d78ac82d81be49bcb3c66c2b7ce6a70b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3f1f4e1449074220843ee2f0e6c1c2aa, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/a3092c9fc13c4b2892d978d59f09ffdd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e3c6e59cb2094bf0b62e9825765d36c4] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=70.6 K 2023-04-03 13:57:09,019 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e6cbcea0d1344bea95b2c29102f60059, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680530218898 2023-04-03 13:57:09,019 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 092b2376d3e047d1af3bef0e9b5fd567, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530220038 2023-04-03 13:57:09,020 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d78ac82d81be49bcb3c66c2b7ce6a70b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680530222191 2023-04-03 13:57:09,020 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 3f1f4e1449074220843ee2f0e6c1c2aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680530222514 2023-04-03 13:57:09,020 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting a3092c9fc13c4b2892d978d59f09ffdd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1680530224650 2023-04-03 13:57:09,020 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e3c6e59cb2094bf0b62e9825765d36c4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680530225780 2023-04-03 13:57:09,030 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/88ee1415975542caaad6b8480f5d14f5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/88ee1415975542caaad6b8480f5d14f5 2023-04-03 13:57:09,033 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#C#compaction#135 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:09,034 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304031b9c25d74c854f23b33278c79dfe9662 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031b9c25d74c854f23b33278c79dfe9662 2023-04-03 13:57:09,038 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/A of 67b1fdef2f316cba9760895849ff9624 into 88ee1415975542caaad6b8480f5d14f5(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:09,038 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:09,038 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/A, priority=10, startTime=1680530228561; duration=0sec 2023-04-03 13:57:09,038 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:09,045 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=219, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/999b41ed31e344f3b6e7771e2b9e7f1c 2023-04-03 13:57:09,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:09,058 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:09,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41090 deadline: 1680530289077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530289078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,084 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/2927edd50abe49dd832156c7604a4294 2023-04-03 13:57:09,100 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/f1ffdf339e13438194720c46a8a12c54 2023-04-03 13:57:09,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/999b41ed31e344f3b6e7771e2b9e7f1c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/999b41ed31e344f3b6e7771e2b9e7f1c 2023-04-03 13:57:09,109 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/999b41ed31e344f3b6e7771e2b9e7f1c, entries=150, sequenceid=219, filesize=25.3 K 2023-04-03 13:57:09,110 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/2927edd50abe49dd832156c7604a4294 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2927edd50abe49dd832156c7604a4294 2023-04-03 13:57:09,116 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2927edd50abe49dd832156c7604a4294, entries=150, sequenceid=219, filesize=11.8 K 2023-04-03 13:57:09,117 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/f1ffdf339e13438194720c46a8a12c54 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1ffdf339e13438194720c46a8a12c54 2023-04-03 13:57:09,122 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1ffdf339e13438194720c46a8a12c54, entries=150, sequenceid=219, filesize=11.8 K 2023-04-03 13:57:09,123 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 67b1fdef2f316cba9760895849ff9624 in 513ms, sequenceid=219, compaction requested=false 2023-04-03 13:57:09,123 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:09,123 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:09,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:09,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:09,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:09,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,125 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:09,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@297666d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:09,126 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:09,126 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,127 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:09,127 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:09,127 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:09,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:09,128 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:09,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,128 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,128 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:09,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:09,132 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:09,132 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:09,132 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:09,132 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:09,132 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:09,133 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,133 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,133 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,134 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,135 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,137 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:09,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:09,138 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:09,138 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:09,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:09,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 13:57:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:09,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:09,196 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403f3d2c8e7896a48b7a749147cdee2486d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f3d2c8e7896a48b7a749147cdee2486d 2023-04-03 13:57:09,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530289205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530289205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,238 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:09,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:09,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:09,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:09,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:09,241 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:09,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:09,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:09,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,242 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,242 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,242 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,242 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:09,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:09,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:09,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:09,243 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,243 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:09,243 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:09,243 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:09,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,244 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:09,244 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:09,244 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,244 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,245 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,245 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:09,246 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:09,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@221a195f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:09,246 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,247 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,247 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:09,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,247 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:09,247 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:09,247 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:09,247 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,247 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,248 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,248 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:09,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:09,250 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:09,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:09,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:09,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:09,250 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:09,251 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,251 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,252 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,258 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,259 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,263 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:09,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:09,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:09,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,264 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,264 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:09,264 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:09,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530289309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530289311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,364 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:09,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:09,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:09,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:09,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:09,367 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:09,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:09,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:09,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:09,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:09,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:09,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:09,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:09,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:09,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:09,369 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,369 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:09,369 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:09,369 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:09,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:09,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:09,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:09,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,371 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,371 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:09,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d37615b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:09,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:09,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,372 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,372 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:09,372 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:09,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:09,372 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:09,373 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:09,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:09,375 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:09,375 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:09,375 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:09,375 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:09,375 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:09,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,377 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,378 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,380 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,380 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,380 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,380 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:09,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:09,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:09,380 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,380 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,380 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,380 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:09,380 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:09,444 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/f1609c01dbbf458688261c3d7e4565cd as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1609c01dbbf458688261c3d7e4565cd 2023-04-03 13:57:09,448 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/C of 67b1fdef2f316cba9760895849ff9624 into f1609c01dbbf458688261c3d7e4565cd(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:09,449 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:09,449 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/C, priority=10, startTime=1680530228561; duration=0sec 2023-04-03 13:57:09,449 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:09,480 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:09,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:09,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:09,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:09,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:09,484 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:09,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:09,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:09,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,485 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,485 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,485 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,486 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:09,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:09,490 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:09,490 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:09,490 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:09,490 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,490 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:09,490 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:09,490 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:09,490 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,490 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:09,491 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:09,491 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:09,491 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:09,502 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,502 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:09,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bfb9e60[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:09,504 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:09,504 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:09,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:09,505 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,505 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:09,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:09,505 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:09,506 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,506 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:09,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:09,508 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:09,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:09,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:09,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:09,508 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:09,509 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,509 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,509 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:09,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,512 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,512 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,512 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:09,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,516 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530289514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530289514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,608 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/be5d0d15710d4a058bf80dc7c639acb0 2023-04-03 13:57:09,612 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:09,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:09,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:09,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:09,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:09,615 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:09,615 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,615 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:09,615 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:09,615 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,615 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,616 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,616 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,616 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,616 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:09,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:09,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:09,617 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:09,617 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,617 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:09,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:09,623 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,623 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:09,623 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:09,624 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:09,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:09,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:09,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:09,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:09,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,630 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,630 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:09,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62662f7c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:09,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:09,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,632 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,632 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:09,632 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:09,632 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:09,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:09,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:09,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:09,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:09,636 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:09,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:09,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:09,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:09,636 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:09,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:09,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:09,643 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:09,643 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:09,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,643 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/1e7d76faa3054affa3628f3ae2ae8ad8 2023-04-03 13:57:09,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,743 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:09,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:09,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:09,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:09,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:09,745 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:09,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:09,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:09,746 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,746 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,746 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,746 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:09,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:09,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:09,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:09,748 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,748 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:09,748 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:09,748 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:09,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:09,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:09,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:09,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:09,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10c4028e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:09,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,753 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:09,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:09,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:09,753 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,753 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:09,754 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:09,754 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,754 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,754 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:09,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:09,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:09,756 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:09,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:09,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:09,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:09,756 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:09,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:09,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:09,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,760 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:09,760 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:09,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:09,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530289817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530289817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,860 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:09,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:09,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:09,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:09,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:09,863 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:09,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:09,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:09,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,864 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,864 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,864 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,864 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:09,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:09,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:09,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:09,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:09,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:09,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:09,865 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,865 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:09,865 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:09,865 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:09,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:09,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,866 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:09,866 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:09,867 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,867 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,867 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:09,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@462f9474[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:09,868 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:09,868 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:09,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:09,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:09,869 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:09,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:09,870 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:09,870 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:09,870 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:09,870 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:09,870 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:09,870 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:09,870 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:09,871 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,871 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,871 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,871 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,872 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,874 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,874 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,874 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:09,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:09,874 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:09,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:09,874 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:09,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,974 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:09,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:09,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:09,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:09,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:09,977 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:09,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:09,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:09,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:09,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:09,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:09,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:09,980 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,980 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:09,980 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:09,980 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:09,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:09,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:09,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41200a4f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:09,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:09,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,983 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:09,983 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:09,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:09,984 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,984 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:09,984 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,985 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:09,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:09,987 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:09,987 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:09,987 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:09,987 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:09,987 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:09,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:09,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:09,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:09,990 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,991 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:09,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:09,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:09,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,996 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:09,996 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:09,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:09,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:10,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/c4f78de6fee547fe9ad97e3bdaabfa20 2023-04-03 13:57:10,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/be5d0d15710d4a058bf80dc7c639acb0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/be5d0d15710d4a058bf80dc7c639acb0 2023-04-03 13:57:10,061 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/be5d0d15710d4a058bf80dc7c639acb0, entries=250, sequenceid=237, filesize=38.9 K 2023-04-03 13:57:10,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/1e7d76faa3054affa3628f3ae2ae8ad8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1e7d76faa3054affa3628f3ae2ae8ad8 2023-04-03 13:57:10,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1e7d76faa3054affa3628f3ae2ae8ad8, entries=150, sequenceid=237, filesize=11.8 K 2023-04-03 13:57:10,067 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/c4f78de6fee547fe9ad97e3bdaabfa20 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/c4f78de6fee547fe9ad97e3bdaabfa20 2023-04-03 13:57:10,074 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/c4f78de6fee547fe9ad97e3bdaabfa20, entries=150, sequenceid=237, filesize=11.8 K 2023-04-03 13:57:10,075 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 67b1fdef2f316cba9760895849ff9624 in 893ms, sequenceid=237, compaction requested=true 2023-04-03 13:57:10,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:10,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:10,075 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:10,075 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:10,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:10,075 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:10,076 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92116 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:10,076 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/A is initiating minor compaction (all files) 2023-04-03 13:57:10,076 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/A in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,076 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/88ee1415975542caaad6b8480f5d14f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/999b41ed31e344f3b6e7771e2b9e7f1c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/be5d0d15710d4a058bf80dc7c639acb0] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=90.0 K 2023-04-03 13:57:10,076 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:10,076 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/B is initiating minor compaction (all files) 2023-04-03 13:57:10,076 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/B in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,076 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1ff076284c444b4ba994b95b44293283, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2927edd50abe49dd832156c7604a4294, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1e7d76faa3054affa3628f3ae2ae8ad8] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=35.7 K 2023-04-03 13:57:10,077 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1ff076284c444b4ba994b95b44293283, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680530225780 2023-04-03 13:57:10,077 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 88ee1415975542caaad6b8480f5d14f5, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680530225780 2023-04-03 13:57:10,078 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2927edd50abe49dd832156c7604a4294, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680530226924 2023-04-03 13:57:10,078 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 999b41ed31e344f3b6e7771e2b9e7f1c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680530226924 2023-04-03 13:57:10,078 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1e7d76faa3054affa3628f3ae2ae8ad8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680530229077 2023-04-03 13:57:10,078 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting be5d0d15710d4a058bf80dc7c639acb0, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680530229065 2023-04-03 13:57:10,088 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#B#compaction#141 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:10,097 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:10,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:10,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:10,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:10,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:10,102 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:10,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:10,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:10,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:10,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,103 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:10,103 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:10,103 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,103 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,104 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#A#compaction#142 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:10,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:10,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:10,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:10,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:10,105 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:10,105 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:10,105 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:10,105 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:10,105 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:10,105 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,105 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:10,105 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 13:57:10,106 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:10,106 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:10,106 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:10,106 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:10,106 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:10,106 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:10,125 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/5870c2a52de74833878c7c2a033bc170 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5870c2a52de74833878c7c2a033bc170 2023-04-03 13:57:10,130 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/B of 67b1fdef2f316cba9760895849ff9624 into 5870c2a52de74833878c7c2a033bc170(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:10,130 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:10,130 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/B, priority=13, startTime=1680530230075; duration=0sec 2023-04-03 13:57:10,130 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:10,130 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:10,131 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:10,131 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/C is initiating minor compaction (all files) 2023-04-03 13:57:10,132 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/C in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,132 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1609c01dbbf458688261c3d7e4565cd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1ffdf339e13438194720c46a8a12c54, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/c4f78de6fee547fe9ad97e3bdaabfa20] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=35.7 K 2023-04-03 13:57:10,132 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f1609c01dbbf458688261c3d7e4565cd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680530225780 2023-04-03 13:57:10,132 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f1ffdf339e13438194720c46a8a12c54, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680530226924 2023-04-03 13:57:10,133 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c4f78de6fee547fe9ad97e3bdaabfa20, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680530229077 2023-04-03 13:57:10,139 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#C#compaction#144 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:10,139 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403645f3a262ce1408493025d57f4fcec1a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403645f3a262ce1408493025d57f4fcec1a 2023-04-03 13:57:10,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:10,323 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:10,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530290339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530290340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530290443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530290443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,548 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/670d687a12324ae7a1e9583010c92312 2023-04-03 13:57:10,558 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/fcc6d5b4d86045d58d8f3743a24bf781 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/fcc6d5b4d86045d58d8f3743a24bf781 2023-04-03 13:57:10,563 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/C of 67b1fdef2f316cba9760895849ff9624 into fcc6d5b4d86045d58d8f3743a24bf781(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:10,563 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:10,563 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/C, priority=13, startTime=1680530230075; duration=0sec 2023-04-03 13:57:10,564 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:10,569 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/72fd979837884cf882e72e246700bc24 2023-04-03 13:57:10,597 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/769f7b5ce0f044ec8e12ffb9a8c5dbf1 2023-04-03 13:57:10,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/670d687a12324ae7a1e9583010c92312 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/670d687a12324ae7a1e9583010c92312 2023-04-03 13:57:10,609 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/670d687a12324ae7a1e9583010c92312, entries=150, sequenceid=254, filesize=25.3 K 2023-04-03 13:57:10,610 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/72fd979837884cf882e72e246700bc24 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/72fd979837884cf882e72e246700bc24 2023-04-03 13:57:10,629 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/72fd979837884cf882e72e246700bc24, entries=150, sequenceid=254, filesize=11.8 K 2023-04-03 13:57:10,630 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/769f7b5ce0f044ec8e12ffb9a8c5dbf1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/769f7b5ce0f044ec8e12ffb9a8c5dbf1 2023-04-03 13:57:10,635 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/769f7b5ce0f044ec8e12ffb9a8c5dbf1, entries=150, sequenceid=254, filesize=11.8 K 2023-04-03 13:57:10,636 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 67b1fdef2f316cba9760895849ff9624 in 531ms, sequenceid=254, compaction requested=false 2023-04-03 13:57:10,636 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:10,636 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,636 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:10,636 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:10,636 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,636 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:10,636 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:10,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:10,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12eed0d8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:10,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:10,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,641 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:10,641 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:10,641 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:10,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:10,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:10,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:10,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:10,645 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:10,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:10,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:10,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:10,645 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:10,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:10,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,649 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:10,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 13:57:10,650 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:10,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:10,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:10,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:10,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:10,650 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:10,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:10,651 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:10,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:10,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:10,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:10,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:10,651 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:10,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:10,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:10,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:10,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530290672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530290675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,751 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:10,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:10,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:10,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:10,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:10,766 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:10,766 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:10,766 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:10,766 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:10,767 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,767 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,768 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:10,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:10,768 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,769 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,769 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:10,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:10,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:10,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:10,775 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:10,775 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:10,775 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:10,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530290776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:10,780 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,780 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:10,780 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:10,780 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:10,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:10,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,786 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,788 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,788 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:10,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:10,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@228e6079[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:10,789 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:10,789 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530290788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,796 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,796 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:10,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:10,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:10,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:10,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:10,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:10,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:10,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:10,805 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:10,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:10,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:10,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:10,805 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:10,813 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,813 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:10,813 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:10,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,815 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,817 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:10,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:10,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:10,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:10,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:10,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:10,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,822 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:10,822 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:10,822 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:10,922 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:10,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:10,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:10,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:10,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:10,927 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:10,927 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:10,927 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:10,927 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:10,927 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,927 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,928 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:10,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:10,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:10,928 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:10,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:10,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:10,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:10,930 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:10,930 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:10,930 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:10,940 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:10,940 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:10,946 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,946 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:10,946 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:10,946 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:10,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:10,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:10,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:10,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:10,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,952 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,955 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/a4fd1cbcce3245e3b8386a2e5f42109a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/a4fd1cbcce3245e3b8386a2e5f42109a 2023-04-03 13:57:10,957 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,957 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:10,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:10,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1838dc68[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:10,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:10,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,960 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/A of 67b1fdef2f316cba9760895849ff9624 into a4fd1cbcce3245e3b8386a2e5f42109a(size=25.8 K), total size for store is 51.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:10,960 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:10,960 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/A, priority=13, startTime=1680530230075; duration=0sec 2023-04-03 13:57:10,961 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:10,963 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:10,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:10,963 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,963 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:10,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:10,964 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:10,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:10,965 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:10,966 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:10,966 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:10,966 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:10,966 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:10,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:10,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:10,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:10,967 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:10,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:10,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:10,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:10,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:10,971 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:10,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:10,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:10,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:10,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:10,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:10,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:10,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:10,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:10,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:10,972 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:10,972 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:10,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530290982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:10,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:10,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530290993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,072 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:11,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:11,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:11,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:11,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:11,078 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:11,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:11,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:11,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:11,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,079 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:11,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:11,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:11,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:11,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:11,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:11,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:11,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:11,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:11,088 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,088 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:11,088 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:11,088 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:11,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:11,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:11,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:11,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,095 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,095 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,095 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,097 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040361348b66c93f4285a5f7359508cabe2e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040361348b66c93f4285a5f7359508cabe2e 2023-04-03 13:57:11,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:11,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590bdf52[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:11,100 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:11,100 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,104 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:11,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:11,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:11,104 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,104 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:11,106 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:11,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:11,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:11,109 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:11,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:11,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:11,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:11,109 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:11,110 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,110 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,110 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:11,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:11,114 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:11,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:11,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:11,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:11,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:11,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,115 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:11,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:11,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:11,115 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:11,125 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=274, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/f84d82294bb04b85ae04b9f1783f6a04 2023-04-03 13:57:11,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/95f44cc741224d569d61b1e78fcc975b 2023-04-03 13:57:11,192 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/1c103c60c2f84ba286dc7559c8e96004 2023-04-03 13:57:11,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/f84d82294bb04b85ae04b9f1783f6a04 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f84d82294bb04b85ae04b9f1783f6a04 2023-04-03 13:57:11,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f84d82294bb04b85ae04b9f1783f6a04, entries=250, sequenceid=274, filesize=39.2 K 2023-04-03 13:57:11,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/95f44cc741224d569d61b1e78fcc975b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/95f44cc741224d569d61b1e78fcc975b 2023-04-03 13:57:11,216 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:11,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:11,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/95f44cc741224d569d61b1e78fcc975b, entries=150, sequenceid=274, filesize=11.9 K 2023-04-03 13:57:11,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/1c103c60c2f84ba286dc7559c8e96004 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/1c103c60c2f84ba286dc7559c8e96004 2023-04-03 13:57:11,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:11,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:11,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:11,225 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:11,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:11,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:11,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:11,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:11,226 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:11,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:11,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:11,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:11,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:11,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:11,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:11,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:11,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:11,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/1c103c60c2f84ba286dc7559c8e96004, entries=150, sequenceid=274, filesize=11.9 K 2023-04-03 13:57:11,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 67b1fdef2f316cba9760895849ff9624 in 585ms, sequenceid=274, compaction requested=true 2023-04-03 13:57:11,234 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:11,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:11,234 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:11,234 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:11,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:11,235 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:11,236 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92468 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:11,236 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/A is initiating minor compaction (all files) 2023-04-03 13:57:11,236 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/A in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,236 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/a4fd1cbcce3245e3b8386a2e5f42109a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/670d687a12324ae7a1e9583010c92312, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f84d82294bb04b85ae04b9f1783f6a04] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=90.3 K 2023-04-03 13:57:11,236 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:11,236 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/B is initiating minor compaction (all files) 2023-04-03 13:57:11,236 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/B in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,236 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5870c2a52de74833878c7c2a033bc170, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/72fd979837884cf882e72e246700bc24, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/95f44cc741224d569d61b1e78fcc975b] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=35.9 K 2023-04-03 13:57:11,237 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5870c2a52de74833878c7c2a033bc170, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680530229077 2023-04-03 13:57:11,237 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting a4fd1cbcce3245e3b8386a2e5f42109a, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680530229077 2023-04-03 13:57:11,238 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 72fd979837884cf882e72e246700bc24, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530229190 2023-04-03 13:57:11,238 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 670d687a12324ae7a1e9583010c92312, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530229190 2023-04-03 13:57:11,239 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 95f44cc741224d569d61b1e78fcc975b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680530230338 2023-04-03 13:57:11,239 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f84d82294bb04b85ae04b9f1783f6a04, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680530230333 2023-04-03 13:57:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:11,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:11,246 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,246 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:11,247 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 13:57:11,247 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:11,247 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:11,247 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:11,248 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:11,248 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:11,248 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:11,259 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#B#compaction#150 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:11,295 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#A#compaction#151 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:11,295 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:11,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:11,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:11,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530291326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:11,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530291331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:11,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530291431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:11,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530291433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530291634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:11,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530291637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,701 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/a54497f574ee487d999272b1669a982c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a54497f574ee487d999272b1669a982c 2023-04-03 13:57:11,706 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/B of 67b1fdef2f316cba9760895849ff9624 into a54497f574ee487d999272b1669a982c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:11,706 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:11,706 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/B, priority=13, startTime=1680530231234; duration=0sec 2023-04-03 13:57:11,707 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:11,707 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:11,708 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:11,708 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/C is initiating minor compaction (all files) 2023-04-03 13:57:11,708 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/C in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,708 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/fcc6d5b4d86045d58d8f3743a24bf781, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/769f7b5ce0f044ec8e12ffb9a8c5dbf1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/1c103c60c2f84ba286dc7559c8e96004] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=35.9 K 2023-04-03 13:57:11,710 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fcc6d5b4d86045d58d8f3743a24bf781, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680530229077 2023-04-03 13:57:11,711 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 769f7b5ce0f044ec8e12ffb9a8c5dbf1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1680530229190 2023-04-03 13:57:11,711 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1c103c60c2f84ba286dc7559c8e96004, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680530230338 2023-04-03 13:57:11,722 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#C#compaction#153 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:11,735 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040300c7d0ffccb1433fa7595c9bdcdb33ef to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040300c7d0ffccb1433fa7595c9bdcdb33ef 2023-04-03 13:57:11,751 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/03431030023f4480b896a8fa05ce25de 2023-04-03 13:57:11,772 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/49445aeafc4945b288ea426b9d746c0c 2023-04-03 13:57:11,782 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/9dab605c34e244f68aed12ab96ff2a87 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9dab605c34e244f68aed12ab96ff2a87 2023-04-03 13:57:11,793 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/A of 67b1fdef2f316cba9760895849ff9624 into 9dab605c34e244f68aed12ab96ff2a87(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:11,793 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:11,793 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/A, priority=13, startTime=1680530231234; duration=0sec 2023-04-03 13:57:11,793 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:11,812 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/477405a28566452fa04c9687a56c5580 2023-04-03 13:57:11,816 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/03431030023f4480b896a8fa05ce25de as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/03431030023f4480b896a8fa05ce25de 2023-04-03 13:57:11,821 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/03431030023f4480b896a8fa05ce25de, entries=150, sequenceid=289, filesize=25.4 K 2023-04-03 13:57:11,823 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/49445aeafc4945b288ea426b9d746c0c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/49445aeafc4945b288ea426b9d746c0c 2023-04-03 13:57:11,829 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/49445aeafc4945b288ea426b9d746c0c, entries=150, sequenceid=289, filesize=11.9 K 2023-04-03 13:57:11,830 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/477405a28566452fa04c9687a56c5580 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/477405a28566452fa04c9687a56c5580 2023-04-03 13:57:11,834 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/477405a28566452fa04c9687a56c5580, entries=150, sequenceid=289, filesize=11.9 K 2023-04-03 13:57:11,835 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 67b1fdef2f316cba9760895849ff9624 in 588ms, sequenceid=289, compaction requested=false 2023-04-03 13:57:11,835 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:11,835 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:11,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:11,836 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,836 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:11,836 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:11,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,839 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,839 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,839 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:11,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d87acbb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:11,840 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:11,840 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,842 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,842 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:11,843 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,843 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:11,843 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:11,843 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:11,843 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,843 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,843 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:11,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:11,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:11,846 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:11,846 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:11,846 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:11,846 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:11,846 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:11,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:11,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:11,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,851 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:11,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:11,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:11,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:11,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:11,851 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:11,851 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:11,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:11,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:11,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,852 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:11,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:11,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:11,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 13:57:11,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:11,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:11,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:11,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:11,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:11,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:11,952 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:11,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:11,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:11,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:11,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:11,958 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403c4b0b33fc1014cd7800cee6cb3a0c179 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c4b0b33fc1014cd7800cee6cb3a0c179 2023-04-03 13:57:11,958 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:11,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:11,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:11,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:11,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,960 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:11,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:11,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:11,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530291958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:11,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:11,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:11,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:11,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:11,962 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:11,962 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:11,962 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,962 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:11,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:11,962 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:11,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530291959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,963 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:11,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:11,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:11,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:11,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:11,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:11,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c5b9eda[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:11,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:11,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,974 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:11,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:11,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:11,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:11,976 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:11,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:11,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:11,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:11,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:11,980 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:11,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:11,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:11,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:11,980 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:11,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:11,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:11,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:11,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,989 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:11,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:11,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,994 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:11,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:11,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:11,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:11,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:11,995 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:11,995 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:11,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:11,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:12,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530292063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:12,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530292064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,095 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:12,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:12,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:12,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:12,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:12,097 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:12,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:12,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:12,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,099 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,099 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,099 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,099 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:12,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:12,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:12,103 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:12,103 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,103 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:12,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:12,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:12,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:12,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:12,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:12,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:12,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:12,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:12,112 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,112 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:12,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325ec4a5[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:12,114 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:12,114 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,119 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:12,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:12,120 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:12,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:12,122 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:12,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:12,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:12,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:12,122 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:12,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,128 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,129 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,130 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/0271d82449cb4120a68f16910fdb5184 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0271d82449cb4120a68f16910fdb5184 2023-04-03 13:57:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,133 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,133 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,134 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,134 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,134 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,134 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,134 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:12,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:12,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,134 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,134 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,134 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,134 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:12,134 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:12,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:12,135 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/C of 67b1fdef2f316cba9760895849ff9624 into 0271d82449cb4120a68f16910fdb5184(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:12,135 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:12,135 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/C, priority=13, startTime=1680530231235; duration=0sec 2023-04-03 13:57:12,135 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:12,234 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:12,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:12,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:12,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:12,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:12,237 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:12,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:12,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:12,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,238 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,238 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,238 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,238 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:12,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:12,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:12,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:12,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:12,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:12,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:12,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:12,239 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:12,239 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,239 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:12,240 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:12,240 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,240 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:12,240 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:12,240 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,240 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:12,240 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:12,240 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,240 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,241 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,241 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:12,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35a87cf3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:12,242 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:12,242 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,242 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:12,242 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:12,242 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:12,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,243 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:12,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:12,245 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:12,245 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:12,245 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:12,245 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:12,245 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:12,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,247 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,248 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,249 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,249 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:12,249 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,249 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:12,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:12,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530292267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:12,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530292267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,349 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:12,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:12,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:12,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:12,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:12,352 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:12,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:12,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:12,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,353 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,353 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,353 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,353 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:12,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:12,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:12,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:12,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:12,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:12,355 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:12,355 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:12,355 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,355 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:12,355 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:12,355 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,355 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:12,355 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:12,355 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,355 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:12,355 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,356 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,356 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,356 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:12,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:12,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35caec89[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:12,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,358 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,358 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:12,358 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:12,358 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:12,358 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,358 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:12,358 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,358 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,359 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,359 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:12,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:12,360 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:12,360 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:12,360 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:12,360 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:12,361 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:12,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,362 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,362 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,364 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,365 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:12,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:12,364 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,365 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,365 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,365 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:12,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,373 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=311, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/532a912d71ed448f9886b1dcd1410de4 2023-04-03 13:57:12,465 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:12,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:12,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:12,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:12,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:12,468 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:12,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:12,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:12,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:12,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:12,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:12,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:12,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:12,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:12,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:12,470 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,470 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:12,470 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:12,471 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:12,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:12,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:12,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:12,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:12,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55719a4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:12,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:12,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,474 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,474 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:12,475 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,475 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:12,475 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:12,475 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:12,476 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,476 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,476 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,476 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:12,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:12,478 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:12,478 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:12,478 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:12,478 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:12,478 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:12,480 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,480 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,480 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,481 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,482 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:12,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,484 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,484 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:12,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530292570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:12,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530292571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,584 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:12,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:12,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:12,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:12,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:12,587 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:12,587 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,587 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:12,587 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:12,587 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,587 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,588 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,588 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,588 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,588 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:12,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:12,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:12,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:12,589 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,589 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:12,589 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:12,589 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:12,589 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:12,590 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,591 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:12,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e89e47[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:12,591 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:12,591 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,592 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:12,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:12,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:12,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:12,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:12,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:12,593 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:12,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:12,595 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:12,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:12,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:12,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:12,595 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:12,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,596 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:12,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:12,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,599 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,599 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,599 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:12,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,599 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:12,699 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:12,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:12,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:12,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:12,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:12,702 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:12,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:12,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:12,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:12,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:12,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:12,704 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,704 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:12,704 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:12,704 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:12,704 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:12,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,705 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:12,706 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:12,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37d13d6d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:12,706 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,707 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,707 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:12,707 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:12,707 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,707 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:12,707 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:12,708 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,708 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:12,708 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,708 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:12,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:12,710 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:12,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:12,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:12,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:12,710 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:12,711 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,711 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,711 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,711 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,712 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,713 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,713 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:12,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,713 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:12,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:12,713 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,713 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,714 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,714 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:12,714 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:12,714 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,714 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,714 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,714 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/4dda7ce06b5943138d764083d973c4bb 2023-04-03 13:57:12,814 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:12,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:12,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:12,816 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:12,816 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,817 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:12,817 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:12,817 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,817 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,817 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,817 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,818 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,818 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:12,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:12,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:12,819 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,819 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:12,819 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:12,819 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:12,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,820 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,820 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,820 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:12,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559d31ba[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:12,821 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:12,821 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,822 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:12,822 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:12,822 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:12,822 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,822 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:12,822 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:12,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,822 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,822 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:12,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:12,824 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:12,824 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:12,824 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:12,824 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:12,824 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:12,825 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,825 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,825 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,825 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,826 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,828 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:12,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,828 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,828 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:12,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,928 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:12,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:12,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:12,931 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:12,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:12,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:12,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,933 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,933 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,933 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:12,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:12,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:12,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:12,934 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,934 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:12,934 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:12,934 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:12,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:12,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,935 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:12,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683a8894[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:12,936 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:12,936 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,937 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:12,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:12,937 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:12,937 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:12,938 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:12,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:12,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:12,940 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:12,940 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:12,940 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:12,940 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:12,940 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:12,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:12,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:12,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:12,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,942 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:12,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:12,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:12,944 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:12,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:12,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:12,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:12,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:12,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:12,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:12,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:12,944 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:12,944 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:12,944 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:12,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:12,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:13,044 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:13,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:13,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:13,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:13,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:13,047 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:13,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:13,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:13,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:13,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:13,048 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:13,048 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,048 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:13,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:13,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:13,049 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:13,049 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:13,049 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:13,049 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:13,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:13,050 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,050 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:13,050 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,050 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:13,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:13,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:13,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d252972[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:13,052 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,052 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:13,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:13,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:13,052 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,052 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:13,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,053 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:13,053 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:13,053 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:13,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:13,055 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:13,055 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:13,055 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:13,055 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:13,055 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:13,055 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,055 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:13,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:13,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,055 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:13,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:13,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,057 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:13,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:13,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:13,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:13,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:13,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:13,058 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:13,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:13,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:13,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,058 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:13,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,059 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:13,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:13,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530293073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:13,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530293075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,159 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:13,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:13,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:13,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:13,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:13,161 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:13,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:13,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:13,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:13,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:13,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:13,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:13,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:13,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:13,164 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:13,164 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:13,164 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:13,164 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:13,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:13,165 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:13,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,165 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:13,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:13,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53a15bee[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:13,167 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:13,167 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,167 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:13,167 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:13,167 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:13,167 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,167 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:13,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:13,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:13,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:13,170 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:13,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:13,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:13,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:13,170 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:13,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:13,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:13,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:13,171 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,172 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:13,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:13,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:13,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:13,174 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:13,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:13,175 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:13,175 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:13,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=311 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/6a9f51bfeade4d0793b10914293e4e18 2023-04-03 13:57:13,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/532a912d71ed448f9886b1dcd1410de4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/532a912d71ed448f9886b1dcd1410de4 2023-04-03 13:57:13,220 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/532a912d71ed448f9886b1dcd1410de4, entries=250, sequenceid=311, filesize=39.2 K 2023-04-03 13:57:13,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/4dda7ce06b5943138d764083d973c4bb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/4dda7ce06b5943138d764083d973c4bb 2023-04-03 13:57:13,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/4dda7ce06b5943138d764083d973c4bb, entries=150, sequenceid=311, filesize=11.9 K 2023-04-03 13:57:13,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/6a9f51bfeade4d0793b10914293e4e18 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/6a9f51bfeade4d0793b10914293e4e18 2023-04-03 13:57:13,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/6a9f51bfeade4d0793b10914293e4e18, entries=150, sequenceid=311, filesize=11.9 K 2023-04-03 13:57:13,230 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 67b1fdef2f316cba9760895849ff9624 in 1289ms, sequenceid=311, compaction requested=true 2023-04-03 13:57:13,230 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:13,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:13,230 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:13,230 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:13,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:13,230 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:13,231 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:13,231 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92870 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:13,231 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/B is initiating minor compaction (all files) 2023-04-03 13:57:13,231 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/A is initiating minor compaction (all files) 2023-04-03 13:57:13,231 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/B in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:13,231 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/A in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:13,231 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a54497f574ee487d999272b1669a982c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/49445aeafc4945b288ea426b9d746c0c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/4dda7ce06b5943138d764083d973c4bb] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=36.3 K 2023-04-03 13:57:13,231 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9dab605c34e244f68aed12ab96ff2a87, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/03431030023f4480b896a8fa05ce25de, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/532a912d71ed448f9886b1dcd1410de4] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=90.7 K 2023-04-03 13:57:13,231 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting a54497f574ee487d999272b1669a982c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680530230338 2023-04-03 13:57:13,231 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9dab605c34e244f68aed12ab96ff2a87, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680530230338 2023-04-03 13:57:13,232 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 49445aeafc4945b288ea426b9d746c0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680530230660 2023-04-03 13:57:13,232 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 03431030023f4480b896a8fa05ce25de, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680530230660 2023-04-03 13:57:13,232 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4dda7ce06b5943138d764083d973c4bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680530231318 2023-04-03 13:57:13,232 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 532a912d71ed448f9886b1dcd1410de4, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680530231318 2023-04-03 13:57:13,244 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#B#compaction#160 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:13,248 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#A#compaction#159 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:13,275 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:13,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:13,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:13,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:13,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:13,278 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:13,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:13,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:13,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:13,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,279 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:13,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:13,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:13,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:13,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:13,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:13,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:13,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:13,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:13,280 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:13,280 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:13,280 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:57:13,280 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:13,280 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:13,281 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:13,281 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:13,281 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:13,281 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:13,299 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040312e06c7e1bd14d14979b3ae611977daa to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040312e06c7e1bd14d14979b3ae611977daa 2023-04-03 13:57:13,657 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/62e5bea8fb4244109a5a6b56e7a8158c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/62e5bea8fb4244109a5a6b56e7a8158c 2023-04-03 13:57:13,662 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/B of 67b1fdef2f316cba9760895849ff9624 into 62e5bea8fb4244109a5a6b56e7a8158c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:13,662 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:13,662 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/B, priority=13, startTime=1680530233230; duration=0sec 2023-04-03 13:57:13,662 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:13,662 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:13,663 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:13,663 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/C is initiating minor compaction (all files) 2023-04-03 13:57:13,663 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/C in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:13,663 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0271d82449cb4120a68f16910fdb5184, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/477405a28566452fa04c9687a56c5580, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/6a9f51bfeade4d0793b10914293e4e18] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=36.3 K 2023-04-03 13:57:13,664 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 0271d82449cb4120a68f16910fdb5184, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680530230338 2023-04-03 13:57:13,664 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 477405a28566452fa04c9687a56c5580, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1680530230660 2023-04-03 13:57:13,664 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 6a9f51bfeade4d0793b10914293e4e18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680530231318 2023-04-03 13:57:13,669 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#C#compaction#162 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:13,704 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=324, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/f72bfb3e6a6947b6a35625eee1d9720a 2023-04-03 13:57:14,070 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/b4a436fe309d4a1eb5b406e091efea95 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/b4a436fe309d4a1eb5b406e091efea95 2023-04-03 13:57:14,076 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/A of 67b1fdef2f316cba9760895849ff9624 into b4a436fe309d4a1eb5b406e091efea95(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:14,076 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:14,076 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/A, priority=13, startTime=1680530233230; duration=0sec 2023-04-03 13:57:14,076 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:14,079 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/00956c5c7f7e45edbd3b4ef540c2811a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/00956c5c7f7e45edbd3b4ef540c2811a 2023-04-03 13:57:14,083 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/C of 67b1fdef2f316cba9760895849ff9624 into 00956c5c7f7e45edbd3b4ef540c2811a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:14,083 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:14,084 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/C, priority=13, startTime=1680530233230; duration=0sec 2023-04-03 13:57:14,084 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:14,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:14,086 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:14,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530294102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530294104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,115 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/6a5222d8ce484dce9de168befe68c097 2023-04-03 13:57:14,124 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/8fac88ff9d264c78be6eece3839058c4 2023-04-03 13:57:14,128 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/f72bfb3e6a6947b6a35625eee1d9720a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f72bfb3e6a6947b6a35625eee1d9720a 2023-04-03 13:57:14,132 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f72bfb3e6a6947b6a35625eee1d9720a, entries=150, sequenceid=324, filesize=25.4 K 2023-04-03 13:57:14,132 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/6a5222d8ce484dce9de168befe68c097 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/6a5222d8ce484dce9de168befe68c097 2023-04-03 13:57:14,136 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/6a5222d8ce484dce9de168befe68c097, entries=150, sequenceid=324, filesize=11.9 K 2023-04-03 13:57:14,137 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/8fac88ff9d264c78be6eece3839058c4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/8fac88ff9d264c78be6eece3839058c4 2023-04-03 13:57:14,140 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/8fac88ff9d264c78be6eece3839058c4, entries=150, sequenceid=324, filesize=11.9 K 2023-04-03 13:57:14,141 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 67b1fdef2f316cba9760895849ff9624 in 861ms, sequenceid=324, compaction requested=false 2023-04-03 13:57:14,141 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:14,141 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,141 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:14,141 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:14,141 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,141 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:14,142 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:14,143 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,143 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,143 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,143 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:14,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8da0c1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:14,145 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:14,145 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,145 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,145 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:14,145 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:14,145 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:14,146 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,146 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:14,146 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:14,146 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,146 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,147 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:14,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:14,149 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:14,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:14,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:14,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:14,149 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:14,150 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,150 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,150 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,151 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,155 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:14,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,155 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,155 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:14,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:14,207 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-03 13:57:14,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:14,207 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:14,207 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:14,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:14,208 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:14,208 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:14,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530294216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,220 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530294217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,256 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:14,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:14,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:14,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:14,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:14,258 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:14,258 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:14,258 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:14,258 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:14,259 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,259 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,259 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,259 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,260 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,260 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:14,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:14,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:14,269 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:14,269 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:14,269 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:14,270 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:14,270 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:14,270 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,270 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:14,270 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:14,270 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,270 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:14,270 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:14,270 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,270 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:14,270 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:14,271 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,271 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,271 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,271 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:14,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:14,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53cffb8c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:14,273 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:14,273 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,273 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,273 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:14,273 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:14,273 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:14,273 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,273 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:14,274 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:14,274 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,274 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:14,279 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:14,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:14,282 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:14,282 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:14,282 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:14,282 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:14,282 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:14,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,290 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,294 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,294 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,294 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:14,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:14,294 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,295 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:14,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,294 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:14,295 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:14,295 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,295 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,295 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,295 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,295 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530294320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530294321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,395 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:14,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:14,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:14,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:14,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:14,397 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:14,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:14,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:14,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:14,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,399 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,399 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,399 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:14,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:14,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:14,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:14,400 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:14,400 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:14,400 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:14,401 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,401 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:14,401 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:14,401 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:14,401 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:14,403 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,403 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,403 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,403 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:14,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1765263e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:14,405 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:14,405 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:14,406 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,406 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:14,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:14,407 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,407 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,407 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,407 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:14,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:14,409 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:14,409 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:14,409 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:14,409 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:14,409 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:14,410 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,410 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,410 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,411 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,412 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:14,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:14,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,413 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,414 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:14,414 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:14,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,514 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:14,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:14,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:14,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:14,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:14,516 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:14,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:14,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:14,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:14,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,517 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:14,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:14,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:14,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:14,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:14,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:14,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:14,519 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,519 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:14,519 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:14,519 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:14,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:14,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:14,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:14,520 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,520 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,520 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,520 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:14,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e426117[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:14,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:14,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:14,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:14,522 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,522 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:14,523 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:14,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:14,525 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:14,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:14,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:14,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:14,525 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:14,526 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,526 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,526 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530294523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,527 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530294523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,529 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:14,529 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:14,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:14,529 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:14,529 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:14,621 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304031aa1f9ffe2b44f6f9aa7bc6df3d79f98 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031aa1f9ffe2b44f6f9aa7bc6df3d79f98 2023-04-03 13:57:14,629 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:14,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:14,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:14,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:14,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:14,632 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:14,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:14,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:14,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:14,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,633 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,633 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,633 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:14,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:14,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:14,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:14,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:14,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. started... 2023-04-03 13:57:14,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. as already flushing 2023-04-03 13:57:14,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:14,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,635 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:14,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:14,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ccb5bb6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:14,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,636 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:14,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:14,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:14,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:14,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:14,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:14,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:14,639 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:14,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:14,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:14,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:14,639 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:14,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:14,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:14,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:14,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:14,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:14,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:14,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:14,643 DEBUG [Thread-789] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:14,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:14,643 DEBUG [Thread-789] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:14,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:14,677 DEBUG [Thread-790] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x61ad5d1c to 127.0.0.1:57081 2023-04-03 13:57:14,677 DEBUG [Thread-790] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:14,680 DEBUG [Thread-792] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x486fba63 to 127.0.0.1:57081 2023-04-03 13:57:14,680 DEBUG [Thread-792] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:14,683 DEBUG [Thread-796] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bd79dff to 127.0.0.1:57081 2023-04-03 13:57:14,683 DEBUG [Thread-796] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:14,683 DEBUG [Thread-794] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50feca84 to 127.0.0.1:57081 2023-04-03 13:57:14,683 DEBUG [Thread-794] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:14,743 DEBUG [Thread-789] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:14,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:14,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530294829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:14,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:14,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530294829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:15,027 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/bc428398a3ac4cbb9954ffddd4034679 2023-04-03 13:57:15,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:15,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41090 deadline: 1680530295333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:15,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:15,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41076 deadline: 1680530295333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:15,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:15,336 DEBUG [Thread-785] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:15,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41082 deadline: 1680530295334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:15,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:15,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41102 deadline: 1680530295347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:15,350 DEBUG [Thread-779] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18212 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:15,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:15,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41096 deadline: 1680530295402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:15,406 DEBUG [Thread-781] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18269 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:15,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/369e487d3b98479d8f04c9917be14ba4 2023-04-03 13:57:15,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/d19e1a6a2f59434d84368b6797a68658 2023-04-03 13:57:15,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/bc428398a3ac4cbb9954ffddd4034679 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bc428398a3ac4cbb9954ffddd4034679 2023-04-03 13:57:15,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bc428398a3ac4cbb9954ffddd4034679, entries=200, sequenceid=348, filesize=32.3 K 2023-04-03 13:57:15,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/369e487d3b98479d8f04c9917be14ba4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/369e487d3b98479d8f04c9917be14ba4 2023-04-03 13:57:15,885 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/369e487d3b98479d8f04c9917be14ba4, entries=150, sequenceid=348, filesize=11.9 K 2023-04-03 13:57:15,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/d19e1a6a2f59434d84368b6797a68658 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d19e1a6a2f59434d84368b6797a68658 2023-04-03 13:57:15,889 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d19e1a6a2f59434d84368b6797a68658, entries=150, sequenceid=348, filesize=11.9 K 2023-04-03 13:57:15,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 67b1fdef2f316cba9760895849ff9624 in 1683ms, sequenceid=348, compaction requested=true 2023-04-03 13:57:15,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:15,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:15,890 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:15,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:15,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:15,890 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:15,891 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85928 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:15,891 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/A is initiating minor compaction (all files) 2023-04-03 13:57:15,891 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:15,891 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/A in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:15,891 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/B is initiating minor compaction (all files) 2023-04-03 13:57:15,891 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/b4a436fe309d4a1eb5b406e091efea95, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f72bfb3e6a6947b6a35625eee1d9720a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bc428398a3ac4cbb9954ffddd4034679] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=83.9 K 2023-04-03 13:57:15,891 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/B in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:15,891 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/62e5bea8fb4244109a5a6b56e7a8158c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/6a5222d8ce484dce9de168befe68c097, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/369e487d3b98479d8f04c9917be14ba4] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=36.4 K 2023-04-03 13:57:15,892 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting b4a436fe309d4a1eb5b406e091efea95, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680530231318 2023-04-03 13:57:15,892 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 62e5bea8fb4244109a5a6b56e7a8158c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680530231318 2023-04-03 13:57:15,892 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f72bfb3e6a6947b6a35625eee1d9720a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680530231953 2023-04-03 13:57:15,892 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 6a5222d8ce484dce9de168befe68c097, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680530231953 2023-04-03 13:57:15,892 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bc428398a3ac4cbb9954ffddd4034679, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680530234099 2023-04-03 13:57:15,892 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 369e487d3b98479d8f04c9917be14ba4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680530234099 2023-04-03 13:57:15,899 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#B#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:15,902 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#A#compaction#168 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:16,296 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-03 13:57:16,311 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/9eb5378b1334405cb3935217f090f682 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/9eb5378b1334405cb3935217f090f682 2023-04-03 13:57:16,317 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/B of 67b1fdef2f316cba9760895849ff9624 into 9eb5378b1334405cb3935217f090f682(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:16,317 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:16,317 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/B, priority=13, startTime=1680530235890; duration=0sec 2023-04-03 13:57:16,317 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:16,317 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:16,318 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:16,319 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 67b1fdef2f316cba9760895849ff9624/C is initiating minor compaction (all files) 2023-04-03 13:57:16,319 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67b1fdef2f316cba9760895849ff9624/C in TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:16,319 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/00956c5c7f7e45edbd3b4ef540c2811a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/8fac88ff9d264c78be6eece3839058c4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d19e1a6a2f59434d84368b6797a68658] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp, totalSize=36.4 K 2023-04-03 13:57:16,319 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 00956c5c7f7e45edbd3b4ef540c2811a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=311, earliestPutTs=1680530231318 2023-04-03 13:57:16,320 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 8fac88ff9d264c78be6eece3839058c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680530231953 2023-04-03 13:57:16,320 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting d19e1a6a2f59434d84368b6797a68658, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680530234099 2023-04-03 13:57:16,329 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67b1fdef2f316cba9760895849ff9624#C#compaction#170 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:16,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:16,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:57:16,341 DEBUG [Thread-787] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7acc1387 to 127.0.0.1:57081 2023-04-03 13:57:16,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:16,342 DEBUG [Thread-787] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:16,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:16,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:16,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:16,342 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:16,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:16,344 DEBUG [Thread-783] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d781def to 127.0.0.1:57081 2023-04-03 13:57:16,344 DEBUG [Thread-783] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:16,742 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/3ad09364522c4b9eadef3c7baf8cd5ec as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3ad09364522c4b9eadef3c7baf8cd5ec 2023-04-03 13:57:16,748 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/C of 67b1fdef2f316cba9760895849ff9624 into 3ad09364522c4b9eadef3c7baf8cd5ec(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:16,748 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:16,748 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/C, priority=13, startTime=1680530235890; duration=0sec 2023-04-03 13:57:16,748 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:16,754 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036a77d9f7a5654d5ca143aceb3476b7eb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a77d9f7a5654d5ca143aceb3476b7eb 2023-04-03 13:57:17,135 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/460739e44ae04d4cb8705fe641462d30 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/460739e44ae04d4cb8705fe641462d30 2023-04-03 13:57:17,143 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67b1fdef2f316cba9760895849ff9624/A of 67b1fdef2f316cba9760895849ff9624 into 460739e44ae04d4cb8705fe641462d30(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-03 13:57:17,144 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:17,144 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624., storeName=67b1fdef2f316cba9760895849ff9624/A, priority=13, startTime=1680530235890; duration=1sec 2023-04-03 13:57:17,144 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:17,163 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=361, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/10ebb3847c3d4e66992e402887210c15 2023-04-03 13:57:17,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/f04080d5abaf4448b791f75d3e7c7335 2023-04-03 13:57:18,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/e9eafce6bccc4925875572372736d435 2023-04-03 13:57:18,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/10ebb3847c3d4e66992e402887210c15 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10ebb3847c3d4e66992e402887210c15 2023-04-03 13:57:18,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10ebb3847c3d4e66992e402887210c15, entries=150, sequenceid=361, filesize=25.4 K 2023-04-03 13:57:18,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/f04080d5abaf4448b791f75d3e7c7335 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/f04080d5abaf4448b791f75d3e7c7335 2023-04-03 13:57:18,028 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/f04080d5abaf4448b791f75d3e7c7335, entries=150, sequenceid=361, filesize=11.9 K 2023-04-03 13:57:18,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/e9eafce6bccc4925875572372736d435 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e9eafce6bccc4925875572372736d435 2023-04-03 13:57:18,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e9eafce6bccc4925875572372736d435, entries=150, sequenceid=361, filesize=11.9 K 2023-04-03 13:57:18,050 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=6.71 KB/6870 for 67b1fdef2f316cba9760895849ff9624 in 1709ms, sequenceid=361, compaction requested=false 2023-04-03 13:57:18,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:25,367 DEBUG [Thread-779] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x071f9a8a to 127.0.0.1:57081 2023-04-03 13:57:25,367 DEBUG [Thread-779] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:25,378 DEBUG [Thread-785] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5649f410 to 127.0.0.1:57081 2023-04-03 13:57:25,378 DEBUG [Thread-785] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:25,427 DEBUG [Thread-781] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x430c7474 to 127.0.0.1:57081 2023-04-03 13:57:25,427 DEBUG [Thread-781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:25,427 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 13:57:25,428 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-03 13:57:25,428 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-03 13:57:25,428 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 115 2023-04-03 13:57:25,428 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-03 13:57:25,428 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-03 13:57:25,428 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 13:57:25,429 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5558 2023-04-03 13:57:25,429 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5482 2023-04-03 13:57:25,429 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 13:57:25,429 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2351 2023-04-03 13:57:25,429 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 7051 rows 2023-04-03 13:57:25,429 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2334 2023-04-03 13:57:25,429 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 7000 rows 2023-04-03 13:57:25,429 INFO [Listener at localhost.localdomain/43477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 13:57:25,429 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35bef85d to 127.0.0.1:57081 2023-04-03 13:57:25,430 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:25,432 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 13:57:25,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-03 13:57:25,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:25,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 13:57:25,437 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530245437"}]},"ts":"1680530245437"} 2023-04-03 13:57:25,438 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 13:57:25,439 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 13:57:25,439 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, UNASSIGN}] 2023-04-03 13:57:25,440 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, UNASSIGN 2023-04-03 13:57:25,441 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=67b1fdef2f316cba9760895849ff9624, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:25,441 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530245441"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530245441"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530245441"}]},"ts":"1680530245441"} 2023-04-03 13:57:25,442 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:57:25,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 13:57:25,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:25,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 67b1fdef2f316cba9760895849ff9624, disabling compactions & flushes 2023-04-03 13:57:25,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:25,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:25,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. after waiting 0 ms 2023-04-03 13:57:25,596 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:25,596 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 67b1fdef2f316cba9760895849ff9624 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 13:57:25,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=A 2023-04-03 13:57:25,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:25,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=B 2023-04-03 13:57:25,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:25,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67b1fdef2f316cba9760895849ff9624, store=C 2023-04-03 13:57:25,597 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:25,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 13:57:26,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304039c3f95fcb5b84f7bac966fa352c6a72c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039c3f95fcb5b84f7bac966fa352c6a72c 2023-04-03 13:57:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 13:57:26,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=370, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/384ce24344d64c68904538c64b9bc05c 2023-04-03 13:57:26,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 13:57:26,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/3a9bb9c3a5514a37b44e3548ee888c3a 2023-04-03 13:57:27,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/4e19abc595ec4d6b9be6b417b6ec67bb 2023-04-03 13:57:27,292 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/A/384ce24344d64c68904538c64b9bc05c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/384ce24344d64c68904538c64b9bc05c 2023-04-03 13:57:27,298 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/384ce24344d64c68904538c64b9bc05c, entries=100, sequenceid=370, filesize=18.6 K 2023-04-03 13:57:27,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/B/3a9bb9c3a5514a37b44e3548ee888c3a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/3a9bb9c3a5514a37b44e3548ee888c3a 2023-04-03 13:57:27,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/3a9bb9c3a5514a37b44e3548ee888c3a, entries=100, sequenceid=370, filesize=9.5 K 2023-04-03 13:57:27,304 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/.tmp/C/4e19abc595ec4d6b9be6b417b6ec67bb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/4e19abc595ec4d6b9be6b417b6ec67bb 2023-04-03 13:57:27,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/4e19abc595ec4d6b9be6b417b6ec67bb, entries=100, sequenceid=370, filesize=9.5 K 2023-04-03 13:57:27,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 67b1fdef2f316cba9760895849ff9624 in 1714ms, sequenceid=370, compaction requested=true 2023-04-03 13:57:27,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5d66fc1b311f40b7aaab3da0da8a56a0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f94d62b93f284513b601b2f403466fe5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c5018dd164084509bbe4ef7f5af95874, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5c131959cff6492b975bb82a31ec10be, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b8831e9785f44a1922058ba259f7b02, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bb20c68c6f72420b82b9522382450204, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10b6e7cb7d5a41e69a2a530ac92a5353, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b0c898eb9e44404a0229bd7cfeb22cc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/aaeff3addc794d9ead3e6878748f771c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9fd4450872274bc888e1a057021dad36, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9c8a9a47d6b04bb0b0bac5c72bc8cde1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c40b2d984b03427aa35fe5948feceeb3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/8993ff439e144d6388811bacfd87cfbe, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/88ee1415975542caaad6b8480f5d14f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/999b41ed31e344f3b6e7771e2b9e7f1c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/be5d0d15710d4a058bf80dc7c639acb0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/a4fd1cbcce3245e3b8386a2e5f42109a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/670d687a12324ae7a1e9583010c92312, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f84d82294bb04b85ae04b9f1783f6a04, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9dab605c34e244f68aed12ab96ff2a87, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/03431030023f4480b896a8fa05ce25de, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/532a912d71ed448f9886b1dcd1410de4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/b4a436fe309d4a1eb5b406e091efea95, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f72bfb3e6a6947b6a35625eee1d9720a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bc428398a3ac4cbb9954ffddd4034679] to archive 2023-04-03 13:57:27,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:57:27,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5d66fc1b311f40b7aaab3da0da8a56a0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5d66fc1b311f40b7aaab3da0da8a56a0 2023-04-03 13:57:27,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f94d62b93f284513b601b2f403466fe5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f94d62b93f284513b601b2f403466fe5 2023-04-03 13:57:27,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c5018dd164084509bbe4ef7f5af95874 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c5018dd164084509bbe4ef7f5af95874 2023-04-03 13:57:27,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5c131959cff6492b975bb82a31ec10be to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/5c131959cff6492b975bb82a31ec10be 2023-04-03 13:57:27,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b8831e9785f44a1922058ba259f7b02 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b8831e9785f44a1922058ba259f7b02 2023-04-03 13:57:27,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bb20c68c6f72420b82b9522382450204 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bb20c68c6f72420b82b9522382450204 2023-04-03 13:57:27,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10b6e7cb7d5a41e69a2a530ac92a5353 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10b6e7cb7d5a41e69a2a530ac92a5353 2023-04-03 13:57:27,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b0c898eb9e44404a0229bd7cfeb22cc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/0b0c898eb9e44404a0229bd7cfeb22cc 2023-04-03 13:57:27,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/aaeff3addc794d9ead3e6878748f771c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/aaeff3addc794d9ead3e6878748f771c 2023-04-03 13:57:27,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9fd4450872274bc888e1a057021dad36 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9fd4450872274bc888e1a057021dad36 2023-04-03 13:57:27,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9c8a9a47d6b04bb0b0bac5c72bc8cde1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9c8a9a47d6b04bb0b0bac5c72bc8cde1 2023-04-03 13:57:27,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c40b2d984b03427aa35fe5948feceeb3 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/c40b2d984b03427aa35fe5948feceeb3 2023-04-03 13:57:27,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/8993ff439e144d6388811bacfd87cfbe to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/8993ff439e144d6388811bacfd87cfbe 2023-04-03 13:57:27,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/88ee1415975542caaad6b8480f5d14f5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/88ee1415975542caaad6b8480f5d14f5 2023-04-03 13:57:27,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/999b41ed31e344f3b6e7771e2b9e7f1c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/999b41ed31e344f3b6e7771e2b9e7f1c 2023-04-03 13:57:27,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/be5d0d15710d4a058bf80dc7c639acb0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/be5d0d15710d4a058bf80dc7c639acb0 2023-04-03 13:57:27,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/a4fd1cbcce3245e3b8386a2e5f42109a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/a4fd1cbcce3245e3b8386a2e5f42109a 2023-04-03 13:57:27,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/670d687a12324ae7a1e9583010c92312 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/670d687a12324ae7a1e9583010c92312 2023-04-03 13:57:27,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f84d82294bb04b85ae04b9f1783f6a04 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f84d82294bb04b85ae04b9f1783f6a04 2023-04-03 13:57:27,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9dab605c34e244f68aed12ab96ff2a87 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/9dab605c34e244f68aed12ab96ff2a87 2023-04-03 13:57:27,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/03431030023f4480b896a8fa05ce25de to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/03431030023f4480b896a8fa05ce25de 2023-04-03 13:57:27,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/532a912d71ed448f9886b1dcd1410de4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/532a912d71ed448f9886b1dcd1410de4 2023-04-03 13:57:27,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/b4a436fe309d4a1eb5b406e091efea95 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/b4a436fe309d4a1eb5b406e091efea95 2023-04-03 13:57:27,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f72bfb3e6a6947b6a35625eee1d9720a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/f72bfb3e6a6947b6a35625eee1d9720a 2023-04-03 13:57:27,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bc428398a3ac4cbb9954ffddd4034679 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/bc428398a3ac4cbb9954ffddd4034679 2023-04-03 13:57:27,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2515b541ee814bd081832cbdd9fb77d0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fb8ff36826a54dfa9bce78d577aec580, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/c25aa519d0744f1e94a1497cfb95cb17, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e8960e1c3f2646819af9999b1831accf, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/b232854552fe41f88a08ce27cf4a1b6d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e2b688949b22459496a3cd0d9a0558ab, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5be6acae10304ffd813b85b8b3729afe, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/543434ddd407431b80306c3c513a68be, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a89b82e7a4634914b3bc4b71439d379d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/609f5760c3764614abdb1ff97ce344d7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fa6fcce71d494a5a80c8a78caa829b29, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/0890820a2bba44a2b9c80aeba414775c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1ff076284c444b4ba994b95b44293283, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/ce6fb5b571d941ffbd638e56c774b2a3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2927edd50abe49dd832156c7604a4294, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5870c2a52de74833878c7c2a033bc170, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1e7d76faa3054affa3628f3ae2ae8ad8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/72fd979837884cf882e72e246700bc24, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a54497f574ee487d999272b1669a982c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/95f44cc741224d569d61b1e78fcc975b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/49445aeafc4945b288ea426b9d746c0c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/62e5bea8fb4244109a5a6b56e7a8158c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/4dda7ce06b5943138d764083d973c4bb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/6a5222d8ce484dce9de168befe68c097, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/369e487d3b98479d8f04c9917be14ba4] to archive 2023-04-03 13:57:27,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:57:27,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2515b541ee814bd081832cbdd9fb77d0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2515b541ee814bd081832cbdd9fb77d0 2023-04-03 13:57:27,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fb8ff36826a54dfa9bce78d577aec580 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fb8ff36826a54dfa9bce78d577aec580 2023-04-03 13:57:27,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/c25aa519d0744f1e94a1497cfb95cb17 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/c25aa519d0744f1e94a1497cfb95cb17 2023-04-03 13:57:27,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e8960e1c3f2646819af9999b1831accf to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e8960e1c3f2646819af9999b1831accf 2023-04-03 13:57:27,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/b232854552fe41f88a08ce27cf4a1b6d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/b232854552fe41f88a08ce27cf4a1b6d 2023-04-03 13:57:27,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e2b688949b22459496a3cd0d9a0558ab to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/e2b688949b22459496a3cd0d9a0558ab 2023-04-03 13:57:27,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5be6acae10304ffd813b85b8b3729afe to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5be6acae10304ffd813b85b8b3729afe 2023-04-03 13:57:27,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/543434ddd407431b80306c3c513a68be to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/543434ddd407431b80306c3c513a68be 2023-04-03 13:57:27,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a89b82e7a4634914b3bc4b71439d379d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a89b82e7a4634914b3bc4b71439d379d 2023-04-03 13:57:27,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/609f5760c3764614abdb1ff97ce344d7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/609f5760c3764614abdb1ff97ce344d7 2023-04-03 13:57:27,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fa6fcce71d494a5a80c8a78caa829b29 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/fa6fcce71d494a5a80c8a78caa829b29 2023-04-03 13:57:27,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/0890820a2bba44a2b9c80aeba414775c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/0890820a2bba44a2b9c80aeba414775c 2023-04-03 13:57:27,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1ff076284c444b4ba994b95b44293283 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1ff076284c444b4ba994b95b44293283 2023-04-03 13:57:27,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/ce6fb5b571d941ffbd638e56c774b2a3 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/ce6fb5b571d941ffbd638e56c774b2a3 2023-04-03 13:57:27,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2927edd50abe49dd832156c7604a4294 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/2927edd50abe49dd832156c7604a4294 2023-04-03 13:57:27,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5870c2a52de74833878c7c2a033bc170 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/5870c2a52de74833878c7c2a033bc170 2023-04-03 13:57:27,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1e7d76faa3054affa3628f3ae2ae8ad8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/1e7d76faa3054affa3628f3ae2ae8ad8 2023-04-03 13:57:27,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/72fd979837884cf882e72e246700bc24 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/72fd979837884cf882e72e246700bc24 2023-04-03 13:57:27,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a54497f574ee487d999272b1669a982c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/a54497f574ee487d999272b1669a982c 2023-04-03 13:57:27,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/95f44cc741224d569d61b1e78fcc975b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/95f44cc741224d569d61b1e78fcc975b 2023-04-03 13:57:27,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/49445aeafc4945b288ea426b9d746c0c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/49445aeafc4945b288ea426b9d746c0c 2023-04-03 13:57:27,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/62e5bea8fb4244109a5a6b56e7a8158c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/62e5bea8fb4244109a5a6b56e7a8158c 2023-04-03 13:57:27,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/4dda7ce06b5943138d764083d973c4bb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/4dda7ce06b5943138d764083d973c4bb 2023-04-03 13:57:27,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/6a5222d8ce484dce9de168befe68c097 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/6a5222d8ce484dce9de168befe68c097 2023-04-03 13:57:27,388 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/369e487d3b98479d8f04c9917be14ba4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/369e487d3b98479d8f04c9917be14ba4 2023-04-03 13:57:27,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d9c3bff28b114e94be1258b7214482d2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/65a165921a274cc888a3adaa2d2387a3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0098bfe86c384317944cf6e984693ee4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/9b6904ef696947c3bb19593c3b36fe2c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0ee12055cf494bf4a04970fcefc108f9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3140743c25fd4ecaa1725272c116748e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e6cbcea0d1344bea95b2c29102f60059, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/35b0d88c1fe14c42b471f1ef6d684376, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/092b2376d3e047d1af3bef0e9b5fd567, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d78ac82d81be49bcb3c66c2b7ce6a70b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3f1f4e1449074220843ee2f0e6c1c2aa, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/a3092c9fc13c4b2892d978d59f09ffdd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1609c01dbbf458688261c3d7e4565cd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e3c6e59cb2094bf0b62e9825765d36c4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1ffdf339e13438194720c46a8a12c54, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/fcc6d5b4d86045d58d8f3743a24bf781, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/c4f78de6fee547fe9ad97e3bdaabfa20, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/769f7b5ce0f044ec8e12ffb9a8c5dbf1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0271d82449cb4120a68f16910fdb5184, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/1c103c60c2f84ba286dc7559c8e96004, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/477405a28566452fa04c9687a56c5580, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/00956c5c7f7e45edbd3b4ef540c2811a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/6a9f51bfeade4d0793b10914293e4e18, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/8fac88ff9d264c78be6eece3839058c4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d19e1a6a2f59434d84368b6797a68658] to archive 2023-04-03 13:57:27,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:57:27,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d9c3bff28b114e94be1258b7214482d2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d9c3bff28b114e94be1258b7214482d2 2023-04-03 13:57:27,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/65a165921a274cc888a3adaa2d2387a3 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/65a165921a274cc888a3adaa2d2387a3 2023-04-03 13:57:27,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0098bfe86c384317944cf6e984693ee4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0098bfe86c384317944cf6e984693ee4 2023-04-03 13:57:27,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/9b6904ef696947c3bb19593c3b36fe2c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/9b6904ef696947c3bb19593c3b36fe2c 2023-04-03 13:57:27,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0ee12055cf494bf4a04970fcefc108f9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0ee12055cf494bf4a04970fcefc108f9 2023-04-03 13:57:27,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3140743c25fd4ecaa1725272c116748e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3140743c25fd4ecaa1725272c116748e 2023-04-03 13:57:27,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e6cbcea0d1344bea95b2c29102f60059 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e6cbcea0d1344bea95b2c29102f60059 2023-04-03 13:57:27,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/35b0d88c1fe14c42b471f1ef6d684376 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/35b0d88c1fe14c42b471f1ef6d684376 2023-04-03 13:57:27,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/092b2376d3e047d1af3bef0e9b5fd567 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/092b2376d3e047d1af3bef0e9b5fd567 2023-04-03 13:57:27,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d78ac82d81be49bcb3c66c2b7ce6a70b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d78ac82d81be49bcb3c66c2b7ce6a70b 2023-04-03 13:57:27,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3f1f4e1449074220843ee2f0e6c1c2aa to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3f1f4e1449074220843ee2f0e6c1c2aa 2023-04-03 13:57:27,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/a3092c9fc13c4b2892d978d59f09ffdd to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/a3092c9fc13c4b2892d978d59f09ffdd 2023-04-03 13:57:27,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1609c01dbbf458688261c3d7e4565cd to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1609c01dbbf458688261c3d7e4565cd 2023-04-03 13:57:27,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e3c6e59cb2094bf0b62e9825765d36c4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e3c6e59cb2094bf0b62e9825765d36c4 2023-04-03 13:57:27,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1ffdf339e13438194720c46a8a12c54 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/f1ffdf339e13438194720c46a8a12c54 2023-04-03 13:57:27,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/fcc6d5b4d86045d58d8f3743a24bf781 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/fcc6d5b4d86045d58d8f3743a24bf781 2023-04-03 13:57:27,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/c4f78de6fee547fe9ad97e3bdaabfa20 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/c4f78de6fee547fe9ad97e3bdaabfa20 2023-04-03 13:57:27,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/769f7b5ce0f044ec8e12ffb9a8c5dbf1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/769f7b5ce0f044ec8e12ffb9a8c5dbf1 2023-04-03 13:57:27,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0271d82449cb4120a68f16910fdb5184 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/0271d82449cb4120a68f16910fdb5184 2023-04-03 13:57:27,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/1c103c60c2f84ba286dc7559c8e96004 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/1c103c60c2f84ba286dc7559c8e96004 2023-04-03 13:57:27,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/477405a28566452fa04c9687a56c5580 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/477405a28566452fa04c9687a56c5580 2023-04-03 13:57:27,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/00956c5c7f7e45edbd3b4ef540c2811a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/00956c5c7f7e45edbd3b4ef540c2811a 2023-04-03 13:57:27,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/6a9f51bfeade4d0793b10914293e4e18 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/6a9f51bfeade4d0793b10914293e4e18 2023-04-03 13:57:27,417 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/8fac88ff9d264c78be6eece3839058c4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/8fac88ff9d264c78be6eece3839058c4 2023-04-03 13:57:27,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d19e1a6a2f59434d84368b6797a68658 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/d19e1a6a2f59434d84368b6797a68658 2023-04-03 13:57:27,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/recovered.edits/373.seqid, newMaxSeqId=373, maxSeqId=4 2023-04-03 13:57:27,425 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624. 2023-04-03 13:57:27,425 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 67b1fdef2f316cba9760895849ff9624: 2023-04-03 13:57:27,427 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:27,427 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=67b1fdef2f316cba9760895849ff9624, regionState=CLOSED 2023-04-03 13:57:27,427 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530247427"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530247427"}]},"ts":"1680530247427"} 2023-04-03 13:57:27,429 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-03 13:57:27,429 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 67b1fdef2f316cba9760895849ff9624, server=jenkins-hbase20.apache.org,39695,1680530174856 in 1.9860 sec 2023-04-03 13:57:27,431 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-03 13:57:27,431 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=67b1fdef2f316cba9760895849ff9624, UNASSIGN in 1.9900 sec 2023-04-03 13:57:27,431 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530247431"}]},"ts":"1680530247431"} 2023-04-03 13:57:27,432 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 13:57:27,433 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 13:57:27,434 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0010 sec 2023-04-03 13:57:27,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 13:57:27,545 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-03 13:57:27,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-03 13:57:27,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:27,547 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:27,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-03 13:57:27,548 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:27,554 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:27,557 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/recovered.edits] 2023-04-03 13:57:27,562 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10ebb3847c3d4e66992e402887210c15 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/10ebb3847c3d4e66992e402887210c15 2023-04-03 13:57:27,564 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/384ce24344d64c68904538c64b9bc05c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/384ce24344d64c68904538c64b9bc05c 2023-04-03 13:57:27,566 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/460739e44ae04d4cb8705fe641462d30 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/A/460739e44ae04d4cb8705fe641462d30 2023-04-03 13:57:27,570 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/3a9bb9c3a5514a37b44e3548ee888c3a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/3a9bb9c3a5514a37b44e3548ee888c3a 2023-04-03 13:57:27,573 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/9eb5378b1334405cb3935217f090f682 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/9eb5378b1334405cb3935217f090f682 2023-04-03 13:57:27,575 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/f04080d5abaf4448b791f75d3e7c7335 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/B/f04080d5abaf4448b791f75d3e7c7335 2023-04-03 13:57:27,579 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3ad09364522c4b9eadef3c7baf8cd5ec to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/3ad09364522c4b9eadef3c7baf8cd5ec 2023-04-03 13:57:27,581 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/4e19abc595ec4d6b9be6b417b6ec67bb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/4e19abc595ec4d6b9be6b417b6ec67bb 2023-04-03 13:57:27,583 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e9eafce6bccc4925875572372736d435 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/C/e9eafce6bccc4925875572372736d435 2023-04-03 13:57:27,587 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/recovered.edits/373.seqid to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624/recovered.edits/373.seqid 2023-04-03 13:57:27,588 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/67b1fdef2f316cba9760895849ff9624 2023-04-03 13:57:27,588 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:57:27,589 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 13:57:27,590 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 13:57:27,594 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040300c7d0ffccb1433fa7595c9bdcdb33ef to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040300c7d0ffccb1433fa7595c9bdcdb33ef 2023-04-03 13:57:27,595 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403100b57641480446982885197c233643e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403100b57641480446982885197c233643e 2023-04-03 13:57:27,596 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040312e06c7e1bd14d14979b3ae611977daa to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040312e06c7e1bd14d14979b3ae611977daa 2023-04-03 13:57:27,597 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031aa1f9ffe2b44f6f9aa7bc6df3d79f98 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031aa1f9ffe2b44f6f9aa7bc6df3d79f98 2023-04-03 13:57:27,598 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031b9c25d74c854f23b33278c79dfe9662 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031b9c25d74c854f23b33278c79dfe9662 2023-04-03 13:57:27,598 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031be3d5fce39a4ffea7bfa0762a298a74 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031be3d5fce39a4ffea7bfa0762a298a74 2023-04-03 13:57:27,599 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034f72fd27c2a64cb5bdc245620f247989 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034f72fd27c2a64cb5bdc245620f247989 2023-04-03 13:57:27,600 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040361348b66c93f4285a5f7359508cabe2e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040361348b66c93f4285a5f7359508cabe2e 2023-04-03 13:57:27,601 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040363511c8dc9244565b25ca3b85c7e29c6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040363511c8dc9244565b25ca3b85c7e29c6 2023-04-03 13:57:27,602 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403645f3a262ce1408493025d57f4fcec1a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403645f3a262ce1408493025d57f4fcec1a 2023-04-03 13:57:27,603 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040368789b4fdabc4f31945e7edfe0b22845 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040368789b4fdabc4f31945e7edfe0b22845 2023-04-03 13:57:27,604 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a77d9f7a5654d5ca143aceb3476b7eb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a77d9f7a5654d5ca143aceb3476b7eb 2023-04-03 13:57:27,606 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040372aeb99241c641498cac29b77164689a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040372aeb99241c641498cac29b77164689a 2023-04-03 13:57:27,607 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037b2fd8d65f9c4b348c2a0fd37fbcf296 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037b2fd8d65f9c4b348c2a0fd37fbcf296 2023-04-03 13:57:27,608 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039c3f95fcb5b84f7bac966fa352c6a72c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039c3f95fcb5b84f7bac966fa352c6a72c 2023-04-03 13:57:27,609 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b19be94642fa41e7b8b2ba9b31a88b73 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b19be94642fa41e7b8b2ba9b31a88b73 2023-04-03 13:57:27,610 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b4497921fe1a467284ed959c7619dc39 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b4497921fe1a467284ed959c7619dc39 2023-04-03 13:57:27,612 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c4b0b33fc1014cd7800cee6cb3a0c179 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c4b0b33fc1014cd7800cee6cb3a0c179 2023-04-03 13:57:27,613 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403dffd4392091b4b7ab8a9a499b1b94dd8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403dffd4392091b4b7ab8a9a499b1b94dd8 2023-04-03 13:57:27,614 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f3d2c8e7896a48b7a749147cdee2486d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f3d2c8e7896a48b7a749147cdee2486d 2023-04-03 13:57:27,615 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fb48e06eac404b79a047ea6150df0649 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fb48e06eac404b79a047ea6150df0649 2023-04-03 13:57:27,615 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 13:57:27,617 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:27,619 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 13:57:27,620 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 13:57:27,621 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:27,621 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 13:57:27,621 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680530247621"}]},"ts":"9223372036854775807"} 2023-04-03 13:57:27,623 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 13:57:27,623 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 67b1fdef2f316cba9760895849ff9624, NAME => 'TestAcidGuarantees,,1680530211488.67b1fdef2f316cba9760895849ff9624.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 13:57:27,623 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 13:57:27,623 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680530247623"}]},"ts":"9223372036854775807"} 2023-04-03 13:57:27,624 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 13:57:27,625 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:27,626 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 80 msec 2023-04-03 13:57:27,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-03 13:57:27,649 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-03 13:57:27,663 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 230), OpenFileDescriptor=509 (was 548), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=350 (was 273) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=2836 (was 2919) 2023-04-03 13:57:27,676 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=509, MaxFileDescriptor=60000, SystemLoadAverage=350, ProcessCount=166, AvailableMemoryMB=2835 2023-04-03 13:57:27,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:57:27,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:57:27,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:27,681 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 13:57:27,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-03 13:57:27,682 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 13:57:27,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 13:57:27,683 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:27,684 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396 empty. 2023-04-03 13:57:27,684 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:27,684 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:57:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 13:57:27,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 13:57:28,096 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:57:28,101 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 653510e400e69700c7ff33f6b46a9396, NAME => 'TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp 2023-04-03 13:57:28,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 13:57:28,513 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:57:28,513 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 653510e400e69700c7ff33f6b46a9396, disabling compactions & flushes 2023-04-03 13:57:28,513 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:28,513 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:28,513 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. after waiting 0 ms 2023-04-03 13:57:28,513 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:28,514 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:28,514 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:28,520 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 13:57:28,523 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530248523"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530248523"}]},"ts":"1680530248523"} 2023-04-03 13:57:28,524 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 13:57:28,525 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 13:57:28,526 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530248525"}]},"ts":"1680530248525"} 2023-04-03 13:57:28,527 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 13:57:28,530 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=653510e400e69700c7ff33f6b46a9396, ASSIGN}] 2023-04-03 13:57:28,532 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=653510e400e69700c7ff33f6b46a9396, ASSIGN 2023-04-03 13:57:28,533 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=653510e400e69700c7ff33f6b46a9396, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=false 2023-04-03 13:57:28,685 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=653510e400e69700c7ff33f6b46a9396, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:28,686 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530248685"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530248685"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530248685"}]},"ts":"1680530248685"} 2023-04-03 13:57:28,689 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:57:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 13:57:28,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:28,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 653510e400e69700c7ff33f6b46a9396, NAME => 'TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:57:28,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:57:28,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,855 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,857 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,859 DEBUG [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A 2023-04-03 13:57:28,859 DEBUG [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A 2023-04-03 13:57:28,859 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:28,859 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 653510e400e69700c7ff33f6b46a9396 columnFamilyName A 2023-04-03 13:57:28,860 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.HStore(310): Store=653510e400e69700c7ff33f6b46a9396/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:28,860 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,862 DEBUG [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B 2023-04-03 13:57:28,862 DEBUG [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B 2023-04-03 13:57:28,862 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:28,862 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 653510e400e69700c7ff33f6b46a9396 columnFamilyName B 2023-04-03 13:57:28,863 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.HStore(310): Store=653510e400e69700c7ff33f6b46a9396/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:28,863 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,864 DEBUG [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C 2023-04-03 13:57:28,864 DEBUG [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C 2023-04-03 13:57:28,865 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:28,865 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 653510e400e69700c7ff33f6b46a9396 columnFamilyName C 2023-04-03 13:57:28,865 INFO [StoreOpener-653510e400e69700c7ff33f6b46a9396-1] regionserver.HStore(310): Store=653510e400e69700c7ff33f6b46a9396/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:28,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:28,866 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,867 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:57:28,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:28,873 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:57:28,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 653510e400e69700c7ff33f6b46a9396; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74101129, jitterRate=0.10419286787509918}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:57:28,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:28,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., pid=30, masterSystemTime=1680530248843 2023-04-03 13:57:28,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:28,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:28,876 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=653510e400e69700c7ff33f6b46a9396, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:28,877 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530248876"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530248876"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530248876"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530248876"}]},"ts":"1680530248876"} 2023-04-03 13:57:28,879 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-03 13:57:28,879 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 in 189 msec 2023-04-03 13:57:28,881 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-03 13:57:28,881 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=653510e400e69700c7ff33f6b46a9396, ASSIGN in 349 msec 2023-04-03 13:57:28,882 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 13:57:28,882 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530248882"}]},"ts":"1680530248882"} 2023-04-03 13:57:28,883 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 13:57:28,885 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 13:57:28,886 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2060 sec 2023-04-03 13:57:29,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 13:57:29,793 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-03 13:57:29,794 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x29efb2d8 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,800 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16367fdc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,803 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,805 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44276, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,806 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 13:57:29,809 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:33278, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 13:57:29,811 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x64e0df67 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,816 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30cdd851, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,817 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x7117e4df to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,822 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@136e067d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,822 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x5c466b6c to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,827 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12394d27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,827 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x46d74611 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,831 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29a548ac, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,832 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x3f837278 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,836 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43184c57, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,837 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x7e3efec6 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,854 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f456fa2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,855 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x6b7f716c to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,861 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62be7a0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,861 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x6ade356f to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,871 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7abacd49, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,871 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x4cbc74e7 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,880 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57d25552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,880 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x3c5d1710 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:29,886 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13695730, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:29,893 DEBUG [hconnection-0x5aefe757-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,893 DEBUG [hconnection-0x5655873c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,893 DEBUG [hconnection-0x73a891b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,894 DEBUG [hconnection-0x31433-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,894 DEBUG [hconnection-0x4c0ff175-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,895 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44302, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:29,895 DEBUG [hconnection-0x2fed28cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,898 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44296, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,898 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44322, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,900 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,901 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,901 DEBUG [hconnection-0x5bd976e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,901 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,901 DEBUG [hconnection-0x45c33c63-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,903 DEBUG [hconnection-0x406ecdf1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,901 DEBUG [hconnection-0xa18ec5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:29,909 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,911 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,911 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,912 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:29,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:29,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:29,912 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:29,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:29,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:29,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:29,918 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:29,918 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:29,921 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:29,921 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:29,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:29,922 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:29,922 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:29,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:29,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:29,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:29,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:29,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:29,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:29,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:29,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:29,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:29,924 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:29,924 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:29,925 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 13:57:29,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:29,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:29,928 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:29,928 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:29,928 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:29,928 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:29,928 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:29,928 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:29,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530309942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:29,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:29,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530309942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:29,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:29,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530309944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:29,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:29,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530309945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:29,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530309945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530310046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530310046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530310048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530310048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530310048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530310248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530310249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530310250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530310251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530310251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,340 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/3a0571c916a149a4993f6ec0d530c133 2023-04-03 13:57:30,362 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/87998b5f9cd346938e90bb930de989d7 2023-04-03 13:57:30,552 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530310551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530310552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530310553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530310553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:30,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530310554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,779 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/a0b2f2e3372741a7aa10ee71e1716592 2023-04-03 13:57:30,784 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/3a0571c916a149a4993f6ec0d530c133 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3a0571c916a149a4993f6ec0d530c133 2023-04-03 13:57:30,788 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3a0571c916a149a4993f6ec0d530c133, entries=150, sequenceid=14, filesize=11.6 K 2023-04-03 13:57:30,789 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/87998b5f9cd346938e90bb930de989d7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/87998b5f9cd346938e90bb930de989d7 2023-04-03 13:57:30,793 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/87998b5f9cd346938e90bb930de989d7, entries=150, sequenceid=14, filesize=11.6 K 2023-04-03 13:57:30,794 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/a0b2f2e3372741a7aa10ee71e1716592 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/a0b2f2e3372741a7aa10ee71e1716592 2023-04-03 13:57:30,797 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/a0b2f2e3372741a7aa10ee71e1716592, entries=150, sequenceid=14, filesize=11.6 K 2023-04-03 13:57:30,798 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 653510e400e69700c7ff33f6b46a9396 in 873ms, sequenceid=14, compaction requested=false 2023-04-03 13:57:30,798 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-03 13:57:30,798 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:30,798 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:30,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:30,799 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:30,799 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,799 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:30,799 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:30,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:30,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:30,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:30,804 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:30,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f4ca38[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:30,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,805 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:30,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:30,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:30,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:30,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:30,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:30,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:30,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:30,808 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:30,808 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:30,808 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:30,808 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:30,808 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:30,809 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,809 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,809 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:30,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:30,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:30,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:30,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,811 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:30,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,812 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:30,812 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:30,812 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:30,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:30,812 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:30,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:30,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:30,812 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:30,813 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,813 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,813 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:30,813 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:30,812 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:30,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:30,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:30,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:30,813 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,813 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,913 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:30,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:30,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:30,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:30,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:30,915 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:30,915 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:30,915 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:30,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:30,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:30,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:30,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:30,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:30,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:30,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:30,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:30,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:30,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:30,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:30,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:30,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:30,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:30,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:30,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:30,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:30,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:30,918 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 13:57:30,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:30,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:30,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:30,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:30,919 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:30,919 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:30,925 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/70f5d613d9414fe198997cf65bfbb8fc 2023-04-03 13:57:30,951 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/1a315fe584284c64b74ee2cc86e43676 2023-04-03 13:57:30,960 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/5a9ba5181d7c4a3c88805363948e995e 2023-04-03 13:57:30,965 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/70f5d613d9414fe198997cf65bfbb8fc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/70f5d613d9414fe198997cf65bfbb8fc 2023-04-03 13:57:30,968 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/70f5d613d9414fe198997cf65bfbb8fc, entries=150, sequenceid=34, filesize=11.6 K 2023-04-03 13:57:30,969 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/1a315fe584284c64b74ee2cc86e43676 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1a315fe584284c64b74ee2cc86e43676 2023-04-03 13:57:30,973 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1a315fe584284c64b74ee2cc86e43676, entries=150, sequenceid=34, filesize=11.6 K 2023-04-03 13:57:30,974 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/5a9ba5181d7c4a3c88805363948e995e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5a9ba5181d7c4a3c88805363948e995e 2023-04-03 13:57:30,979 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5a9ba5181d7c4a3c88805363948e995e, entries=150, sequenceid=34, filesize=11.6 K 2023-04-03 13:57:30,980 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 653510e400e69700c7ff33f6b46a9396 in 62ms, sequenceid=34, compaction requested=false 2023-04-03 13:57:30,980 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:30,980 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:30,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:30,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:30,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:30,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:30,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:30,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:30,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:30,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:30,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e1e54ce[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:30,984 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:30,984 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,985 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,985 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:30,985 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:30,985 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:30,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:30,989 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:30,989 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,989 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:30,989 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:30,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:30,992 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:30,992 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:30,993 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:30,993 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:30,993 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:30,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:30,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:30,994 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:30,998 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:30,998 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:30,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:30,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:30,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:30,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:30,998 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:30,999 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:31,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:31,061 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:57:31,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:31,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:31,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:31,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:31,062 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:31,062 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:31,099 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:31,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530251074, max timestamp=1680530251097Num uniques 859; 2023-04-03 13:57:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530251074, max timestamp=1680530251097Num uniques 859; 2023-04-03 13:57:31,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:31,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530251074, max timestamp=1680530251097Num uniques 859; 2023-04-03 13:57:31,100 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:31,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:31,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:31,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:31,101 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:31,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:31,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:31,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,102 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,103 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,103 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:31,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:31,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:31,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:31,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:31,104 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:31,104 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:31,105 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,105 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,106 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:31,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@389ccc3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:31,107 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:31,107 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:31,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:31,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:31,107 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,108 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:31,108 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,108 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,108 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,108 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:31,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:31,110 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:31,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:31,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:31,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:31,110 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:31,111 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,111 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,111 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,113 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530311111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,115 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:31,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:31,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,116 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:31,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,116 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:31,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530311112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530311112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530311115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530311115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,216 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:31,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:31,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530311216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530311217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:31,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:31,219 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:31,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:31,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:31,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530311217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,220 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:31,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:31,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:31,221 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,221 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:31,221 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:31,221 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:31,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:31,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,222 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530311219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,222 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530311220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,223 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,223 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:31,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@150921ef[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:31,224 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:31,224 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,224 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:31,224 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:31,224 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:31,224 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,224 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:31,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,226 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:31,226 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,226 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:31,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:31,227 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:31,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:31,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:31,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:31,227 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:31,228 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,228 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,228 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,229 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,230 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:31,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,232 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:31,232 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:31,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,332 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:31,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:31,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:31,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:31,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:31,335 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:31,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:31,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:31,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,336 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:31,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:31,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:31,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:31,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:31,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:31,338 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,338 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:31,338 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:31,338 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:31,339 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,339 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,339 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,339 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:31,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53ea8c7c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:31,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:31,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,341 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:31,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:31,341 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,341 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:31,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:31,341 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,342 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,342 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:31,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:31,343 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:31,343 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:31,343 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:31,343 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:31,344 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:31,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,345 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,348 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:31,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:31,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,348 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:31,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,348 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:31,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530311419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530311419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530311421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530311423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530311424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,448 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:31,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:31,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:31,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:31,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:31,451 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:31,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:31,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:31,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,452 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,452 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,452 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:31,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:31,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:31,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:31,453 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:31,453 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,453 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:31,453 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:31,453 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:31,453 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,453 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:31,453 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:31,453 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:31,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:31,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:31,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:31,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,468 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:31,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ffe1aeb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:31,470 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:31,470 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:31,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:31,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:31,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:31,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:31,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/2e9d0162ca834c9f8f4cc81750d16366 2023-04-03 13:57:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:31,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:31,480 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:31,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:31,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:31,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:31,480 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:31,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,486 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,487 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,488 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,488 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:31,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:31,489 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:31,489 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:31,489 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:31,489 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,489 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,589 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:31,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:31,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:31,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:31,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:31,591 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:31,591 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,591 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:31,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:31,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,592 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,592 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,593 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,593 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:31,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:31,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:31,594 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:31,594 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,594 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:31,594 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:31,594 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:31,594 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,594 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:31,594 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:31,594 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,594 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:31,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:31,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:31,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:31,595 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,596 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,596 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:31,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:31,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32dc3e03[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:31,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,597 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:31,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:31,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:31,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,599 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,599 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:31,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:31,601 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:31,601 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:31,601 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:31,601 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:31,601 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:31,602 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,602 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,602 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,604 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,606 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,606 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,606 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,607 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:31,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:31,606 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,607 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:31,607 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:31,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:31,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,707 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:31,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:31,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:31,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:31,710 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:31,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:31,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:31,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,710 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,711 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,711 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,711 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,711 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:31,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:31,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:31,712 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:31,713 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,713 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:31,713 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:31,713 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:31,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,714 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:31,716 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:31,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8667ee[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:31,716 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,716 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:31,717 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,717 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:31,717 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,717 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:31,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:31,719 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:31,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:31,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:31,719 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:31,719 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:31,720 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,720 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,720 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,721 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,722 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:31,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:31,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,724 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530311723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,724 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:31,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:31,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530311723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,724 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:31,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530311725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530311725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:31,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530311727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,824 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:31,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:31,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:31,827 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:31,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:31,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:31,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,828 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,828 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,828 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:31,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:31,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:31,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:31,829 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:31,829 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:31,829 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:31,830 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:31,830 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,830 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69088f2e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:31,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:31,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,832 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:31,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:31,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:31,833 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,833 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:31,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:31,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:31,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:31,835 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:31,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:31,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:31,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:31,835 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:31,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,836 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,837 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,839 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:31,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:31,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,839 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:31,839 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/4ecd21bff44749caa90e6767abc7239f 2023-04-03 13:57:31,939 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:31,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:31,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:31,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:31,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:31,942 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:31,942 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,942 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:31,942 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:31,942 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,942 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,943 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,943 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,943 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:31,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:31,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:31,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:31,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:31,944 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:31,944 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:31,944 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:31,944 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:31,944 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:31,944 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,945 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:31,945 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:31,945 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:31,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:31,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:31,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:31,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:31,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,946 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:31,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:31,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@662ee9a6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:31,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,949 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:31,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:31,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:31,949 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,949 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:31,949 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,950 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,950 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:31,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:31,951 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:31,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:31,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:31,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:31,952 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:31,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:31,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:31,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:31,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,953 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,954 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:31,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:31,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:31,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:31,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:31,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:31,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:31,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:31,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:31,956 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:31,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:31,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:31,956 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:31,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:31,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:31,956 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:32,056 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:32,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:32,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:32,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:32,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:32,059 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:32,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:32,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:32,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:32,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,061 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:32,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:32,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:32,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:32,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:32,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:32,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:32,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:32,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:32,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:32,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:32,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:32,062 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,063 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:32,063 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:32,063 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:32,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:32,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:32,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:32,063 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,064 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:32,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35d1ef46[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:32,065 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:32,065 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,065 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:32,066 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,066 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:32,066 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,067 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:32,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:32,069 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:32,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:32,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:32,069 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:32,069 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:32,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:32,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,070 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,071 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,073 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:32,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:32,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:32,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:32,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:32,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:32,073 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:32,073 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:32,173 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:32,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:32,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:32,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:32,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:32,176 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:32,176 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:32,176 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:32,176 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:32,176 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,176 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:32,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:32,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:32,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:32,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:32,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:32,185 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,185 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:32,185 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:32,185 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:32,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:32,186 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:32,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68355705[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:32,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:32,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,187 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:32,188 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,188 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:32,188 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:32,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:32,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:32,190 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:32,190 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:32,190 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:32,190 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:32,190 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:32,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,191 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:32,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:32,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:32,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:32,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:32,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:32,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,194 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,194 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:32,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:32,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530312228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:32,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530312229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:32,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530312229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:32,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530312230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:32,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530312232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,294 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:32,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:32,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:32,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:32,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:32,297 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:32,297 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:32,297 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:32,297 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:32,298 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,298 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,298 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:32,298 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:32,298 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,299 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:32,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:32,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:32,300 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,300 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:32,300 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:32,300 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:32,300 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:32,301 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,301 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,301 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,301 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:32,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4569c4d2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:32,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:32,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,303 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,303 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:32,303 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:32,303 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:32,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:32,304 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f8aeba027b8745469c7425302deb3989 2023-04-03 13:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:32,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:32,306 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:32,306 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:32,306 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:32,306 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:32,306 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:32,307 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,307 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,307 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:32,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:32,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:32,308 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,309 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/2e9d0162ca834c9f8f4cc81750d16366 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2e9d0162ca834c9f8f4cc81750d16366 2023-04-03 13:57:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,310 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:32,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:32,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:32,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:32,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:32,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:32,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:32,312 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:32,312 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:32,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:32,312 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:32,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:32,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:32,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:32,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2e9d0162ca834c9f8f4cc81750d16366, entries=100, sequenceid=45, filesize=9.3 K 2023-04-03 13:57:32,314 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/4ecd21bff44749caa90e6767abc7239f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4ecd21bff44749caa90e6767abc7239f 2023-04-03 13:57:32,317 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4ecd21bff44749caa90e6767abc7239f, entries=100, sequenceid=45, filesize=9.3 K 2023-04-03 13:57:32,318 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f8aeba027b8745469c7425302deb3989 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8aeba027b8745469c7425302deb3989 2023-04-03 13:57:32,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8aeba027b8745469c7425302deb3989, entries=100, sequenceid=45, filesize=9.3 K 2023-04-03 13:57:32,326 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 653510e400e69700c7ff33f6b46a9396 in 1265ms, sequenceid=45, compaction requested=true 2023-04-03 13:57:32,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:32,327 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:32,327 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:32,328 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:32,328 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:32,328 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:32,328 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,328 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:32,328 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,328 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3a0571c916a149a4993f6ec0d530c133, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/70f5d613d9414fe198997cf65bfbb8fc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2e9d0162ca834c9f8f4cc81750d16366] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=32.5 K 2023-04-03 13:57:32,329 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/87998b5f9cd346938e90bb930de989d7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1a315fe584284c64b74ee2cc86e43676, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4ecd21bff44749caa90e6767abc7239f] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=32.5 K 2023-04-03 13:57:32,329 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3a0571c916a149a4993f6ec0d530c133, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680530249923 2023-04-03 13:57:32,329 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 87998b5f9cd346938e90bb930de989d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680530249923 2023-04-03 13:57:32,329 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 70f5d613d9414fe198997cf65bfbb8fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680530249943 2023-04-03 13:57:32,329 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1a315fe584284c64b74ee2cc86e43676, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680530249943 2023-04-03 13:57:32,329 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2e9d0162ca834c9f8f4cc81750d16366, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680530251060 2023-04-03 13:57:32,330 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4ecd21bff44749caa90e6767abc7239f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680530251060 2023-04-03 13:57:32,337 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#186 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:32,339 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#187 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:32,413 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:32,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:32,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:32,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:32,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:32,415 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:32,415 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:32,415 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:32,415 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:32,416 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,416 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,416 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:32,416 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:32,417 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:32,417 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:32,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:32,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:32,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:32,417 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:32,418 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:32,418 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 13:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:32,418 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:32,424 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/06628fdc841d4761b263ac49b8dc943c 2023-04-03 13:57:32,754 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/e0538a985c824d0f85935dbabf669296 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e0538a985c824d0f85935dbabf669296 2023-04-03 13:57:32,758 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/e01877a66f6c4f1e9f9cc44cd9f7cd15 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e01877a66f6c4f1e9f9cc44cd9f7cd15 2023-04-03 13:57:32,762 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into e0538a985c824d0f85935dbabf669296(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:32,762 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:32,762 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=13, startTime=1680530252327; duration=0sec 2023-04-03 13:57:32,762 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:32,762 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:32,763 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:32,763 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:32,763 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:32,763 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/a0b2f2e3372741a7aa10ee71e1716592, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5a9ba5181d7c4a3c88805363948e995e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8aeba027b8745469c7425302deb3989] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=32.5 K 2023-04-03 13:57:32,764 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting a0b2f2e3372741a7aa10ee71e1716592, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680530249923 2023-04-03 13:57:32,764 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5a9ba5181d7c4a3c88805363948e995e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680530249943 2023-04-03 13:57:32,764 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into e01877a66f6c4f1e9f9cc44cd9f7cd15(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:32,764 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f8aeba027b8745469c7425302deb3989, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680530251060 2023-04-03 13:57:32,764 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:32,764 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=13, startTime=1680530252327; duration=0sec 2023-04-03 13:57:32,764 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:32,770 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#190 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:32,877 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/c3ec663965fb4100b95eb34273107db3 2023-04-03 13:57:32,965 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 13:57:33,180 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/c803b35524194c8cbc9081459f3bfa84 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c803b35524194c8cbc9081459f3bfa84 2023-04-03 13:57:33,184 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into c803b35524194c8cbc9081459f3bfa84(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:33,184 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:33,184 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=13, startTime=1680530252327; duration=0sec 2023-04-03 13:57:33,184 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:33,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:33,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:33,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530313237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530313238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530313238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530313239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530313241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,286 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/3cb44f35f8274d29968a0b27a9c2bdc1 2023-04-03 13:57:33,289 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/06628fdc841d4761b263ac49b8dc943c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/06628fdc841d4761b263ac49b8dc943c 2023-04-03 13:57:33,293 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/06628fdc841d4761b263ac49b8dc943c, entries=150, sequenceid=82, filesize=11.6 K 2023-04-03 13:57:33,293 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/c3ec663965fb4100b95eb34273107db3 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c3ec663965fb4100b95eb34273107db3 2023-04-03 13:57:33,299 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c3ec663965fb4100b95eb34273107db3, entries=150, sequenceid=82, filesize=11.6 K 2023-04-03 13:57:33,302 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/3cb44f35f8274d29968a0b27a9c2bdc1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3cb44f35f8274d29968a0b27a9c2bdc1 2023-04-03 13:57:33,308 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3cb44f35f8274d29968a0b27a9c2bdc1, entries=150, sequenceid=82, filesize=11.6 K 2023-04-03 13:57:33,309 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for 653510e400e69700c7ff33f6b46a9396 in 891ms, sequenceid=82, compaction requested=false 2023-04-03 13:57:33,309 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:33,309 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:33,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:33,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:33,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:33,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:33,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@541b6600[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:33,315 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:33,315 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,316 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:33,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:33,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:33,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:33,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:33,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:33,319 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:33,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:33,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:33,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:33,319 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:33,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,321 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,321 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:33,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:33,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,323 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,324 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:33,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,324 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:33,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:33,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 13:57:33,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:33,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:33,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:33,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:33,346 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:33,346 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:33,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530313369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530313370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530313370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530313370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,424 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:33,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:33,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:33,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:33,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:33,427 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:33,427 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,427 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:33,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:33,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,428 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,429 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,429 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:33,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:33,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:33,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:33,430 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,430 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:33,430 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:33,430 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:33,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:33,431 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,432 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,432 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,432 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:33,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:33,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e8a081a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:33,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,433 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:33,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,434 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:33,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:33,436 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:33,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:33,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:33,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:33,437 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:33,437 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,437 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,439 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,440 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,440 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,440 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:33,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:33,440 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,441 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:33,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,441 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:33,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530313473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530313473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530313473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,475 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530313473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,541 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:33,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:33,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:33,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:33,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:33,544 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:33,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:33,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:33,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,545 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,545 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,545 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,545 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:33,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:33,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:33,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:33,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:33,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:33,546 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,547 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:33,547 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:33,547 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,547 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:33,547 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:33,547 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,547 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:33,547 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:33,547 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,547 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,548 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,548 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:33,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c45e7b1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:33,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:33,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:33,550 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:33,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:33,552 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:33,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:33,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:33,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:33,552 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:33,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,553 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,554 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,556 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:33,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:33,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:33,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,556 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:33,556 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:33,656 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:33,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:33,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:33,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:33,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:33,659 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:33,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:33,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:33,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,660 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,660 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,660 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,660 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:33,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:33,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:33,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:33,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:33,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:33,661 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,661 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:33,662 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:33,662 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:33,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:33,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:33,665 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:33,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2781e9bf[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:33,665 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,665 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:33,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:33,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:33,665 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,665 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:33,666 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,667 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:33,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:33,668 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:33,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:33,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:33,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:33,668 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:33,669 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,669 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,669 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,670 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,671 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:33,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:33,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,672 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,672 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:33,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,673 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,673 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,673 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:33,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:33,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530313675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530313675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530313675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530313676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,765 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/3d0cf49404244fb3b940f739234fc620 2023-04-03 13:57:33,773 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:33,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:33,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:33,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:33,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:33,790 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:33,790 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,790 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:33,790 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:33,790 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,791 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,791 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,791 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,792 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,792 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:33,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:33,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:33,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:33,806 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:33,806 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:33,806 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:33,806 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:33,807 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,807 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,807 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:33,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a5a029[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:33,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:33,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,809 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:33,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:33,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:33,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:33,810 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,810 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:33,810 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,810 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:33,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:33,813 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:33,813 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:33,813 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:33,813 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:33,813 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:33,814 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,814 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,814 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,814 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,816 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:33,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:33,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,821 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,821 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:33,821 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:33,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,921 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:33,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:33,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:33,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:33,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:33,924 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:33,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:33,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:33,925 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,925 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,926 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,926 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,926 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,926 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:33,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:33,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:33,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:33,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:33,927 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,927 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:33,927 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:33,927 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:33,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:33,928 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,928 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,929 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,929 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:33,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16d1b7ff[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:33,930 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:33,930 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,931 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,931 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:33,931 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:33,931 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:33,932 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,932 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:33,932 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,933 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:33,933 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,933 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:33,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:33,936 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:33,936 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:33,936 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:33,936 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:33,936 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:33,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:33,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:33,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,939 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:33,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:33,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:33,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:33,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:33,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:33,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:33,941 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:33,942 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:33,942 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:33,942 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:33,942 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:33,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:33,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530313977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530313978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530313979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:33,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:33,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530313979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,042 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:34,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:34,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:34,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:34,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:34,045 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:34,045 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,045 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:34,045 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:34,045 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,045 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,046 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:34,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:34,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:34,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:34,047 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:34,047 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,047 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:34,047 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:34,047 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:34,047 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,047 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:34,048 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:34,048 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:34,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:34,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:34,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:34,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:34,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@152ab2fb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:34,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:34,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,051 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:34,051 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:34,051 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:34,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:34,052 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:34,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:34,054 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:34,054 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:34,054 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:34,054 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:34,054 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:34,055 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,055 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,055 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,055 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:34,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:34,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:34,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,058 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,059 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:34,059 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:34,058 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,059 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,159 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:34,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:34,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:34,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:34,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:34,161 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:34,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:34,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:34,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,163 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:34,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:34,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:34,164 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,164 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:34,164 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:34,164 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:34,165 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,165 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,165 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,165 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:34,166 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:34,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dce6df0[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:34,166 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,167 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,167 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,167 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:34,167 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:34,167 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:34,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:34,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:34,170 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:34,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:34,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:34,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:34,170 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,171 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,172 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:34,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:34,174 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,174 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:34,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,174 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:34,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,175 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,175 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:34,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/4722ee2ecadb4cd3a09cc9afd9cfe11f 2023-04-03 13:57:34,275 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:34,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:34,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:34,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:34,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:34,277 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:34,277 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,277 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:34,277 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:34,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,278 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,279 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:34,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:34,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:34,279 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:34,280 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,280 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:34,280 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:34,280 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:34,281 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,281 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,281 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,281 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:34,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386f2688[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:34,282 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:34,282 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:34,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:34,283 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,283 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:34,283 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:34,284 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,284 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,284 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,284 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:34,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:34,286 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:34,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:34,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:34,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:34,286 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:34,287 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,287 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,287 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,287 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,290 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:34,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,290 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,290 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,290 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,390 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:34,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:34,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:34,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:34,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:34,393 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:34,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:34,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:34,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,394 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:34,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:34,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:34,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:34,395 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,395 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:34,395 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:34,395 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:34,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,396 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,397 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,397 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:34,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:34,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231a0035[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:34,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:34,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:34,399 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:34,399 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,399 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:34,400 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,400 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:34,400 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,400 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:34,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:34,401 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:34,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:34,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:34,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:34,402 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:34,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,403 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,405 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,406 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:34,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:34,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,406 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:34,406 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:34,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:34,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530314483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:34,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:34,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530314484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530314484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:34,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530314486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,506 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:34,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:34,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:34,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:34,509 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:34,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:34,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:34,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,510 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:34,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:34,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:34,512 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,512 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:34,512 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:34,512 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:34,513 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,513 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,513 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:34,514 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:34,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e7d3c7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:34,514 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:34,515 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:34,515 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:34,515 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:34,516 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:34,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:34,518 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:34,518 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:34,518 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:34,518 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:34,518 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:34,518 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,519 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,520 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:34,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:34,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:34,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,522 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:34,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,522 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:34,522 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,522 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,522 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,622 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:34,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:34,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:34,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:34,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:34,625 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:34,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:34,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:34,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,626 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:34,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:34,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:34,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:34,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:34,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:34,627 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:34,627 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:34,628 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:34,628 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:34,628 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=99 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/c012c62ad7314abca8678aae48cc4884 2023-04-03 13:57:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:34,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d0893a6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:34,630 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:34,630 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,632 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:34,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,632 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:34,632 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:34,632 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:34,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,633 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:34,633 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/3d0cf49404244fb3b940f739234fc620 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3d0cf49404244fb3b940f739234fc620 2023-04-03 13:57:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:34,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:34,642 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:34,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:34,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:34,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:34,642 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:34,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:34,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,646 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:34,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:34,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3d0cf49404244fb3b940f739234fc620, entries=150, sequenceid=99, filesize=11.6 K 2023-04-03 13:57:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:34,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/4722ee2ecadb4cd3a09cc9afd9cfe11f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4722ee2ecadb4cd3a09cc9afd9cfe11f 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,654 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:34,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:34,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:34,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:34,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:34,654 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:34,655 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:34,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4722ee2ecadb4cd3a09cc9afd9cfe11f, entries=150, sequenceid=99, filesize=11.6 K 2023-04-03 13:57:34,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/c012c62ad7314abca8678aae48cc4884 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c012c62ad7314abca8678aae48cc4884 2023-04-03 13:57:34,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c012c62ad7314abca8678aae48cc4884, entries=150, sequenceid=99, filesize=11.6 K 2023-04-03 13:57:34,664 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 653510e400e69700c7ff33f6b46a9396 in 1318ms, sequenceid=99, compaction requested=true 2023-04-03 13:57:34,664 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:34,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:34,664 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:34,664 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:34,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:34,664 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:34,665 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:34,665 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:34,665 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,665 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e0538a985c824d0f85935dbabf669296, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/06628fdc841d4761b263ac49b8dc943c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3d0cf49404244fb3b940f739234fc620] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=34.9 K 2023-04-03 13:57:34,666 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:34,666 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:34,666 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting e0538a985c824d0f85935dbabf669296, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680530249943 2023-04-03 13:57:34,666 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,666 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e01877a66f6c4f1e9f9cc44cd9f7cd15, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c3ec663965fb4100b95eb34273107db3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4722ee2ecadb4cd3a09cc9afd9cfe11f] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=34.9 K 2023-04-03 13:57:34,666 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 06628fdc841d4761b263ac49b8dc943c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680530251103 2023-04-03 13:57:34,666 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e01877a66f6c4f1e9f9cc44cd9f7cd15, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680530249943 2023-04-03 13:57:34,666 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3d0cf49404244fb3b940f739234fc620, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680530253343 2023-04-03 13:57:34,667 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c3ec663965fb4100b95eb34273107db3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680530251103 2023-04-03 13:57:34,667 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4722ee2ecadb4cd3a09cc9afd9cfe11f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680530253343 2023-04-03 13:57:34,671 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#196 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:34,672 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#195 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:34,684 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/d23b8cbf94984d5a963b0a9f2720195a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d23b8cbf94984d5a963b0a9f2720195a 2023-04-03 13:57:34,688 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into d23b8cbf94984d5a963b0a9f2720195a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:34,688 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:34,688 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=13, startTime=1680530254664; duration=0sec 2023-04-03 13:57:34,688 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:34,688 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:34,689 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:34,689 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:34,689 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,689 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c803b35524194c8cbc9081459f3bfa84, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3cb44f35f8274d29968a0b27a9c2bdc1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c012c62ad7314abca8678aae48cc4884] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=34.9 K 2023-04-03 13:57:34,689 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c803b35524194c8cbc9081459f3bfa84, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680530249943 2023-04-03 13:57:34,689 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 3cb44f35f8274d29968a0b27a9c2bdc1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680530251103 2023-04-03 13:57:34,689 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c012c62ad7314abca8678aae48cc4884, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680530253343 2023-04-03 13:57:34,707 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#197 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:34,755 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:34,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:34,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:34,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:34,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:34,757 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:34,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:34,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:34,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,758 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:34,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:34,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:34,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:34,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:34,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:34,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:34,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:34,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:34,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:34,760 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:34,760 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:34,760 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:34,760 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:34,760 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 13:57:34,760 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:34,760 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:34,760 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:34,760 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:34,760 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:34,760 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:35,086 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/80f90dd1234d42079dd598f44570754a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/80f90dd1234d42079dd598f44570754a 2023-04-03 13:57:35,089 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into 80f90dd1234d42079dd598f44570754a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:35,089 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:35,089 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=13, startTime=1680530254664; duration=0sec 2023-04-03 13:57:35,090 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:35,115 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/9d0738cb39fc4b5b998f01bb806a69e5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9d0738cb39fc4b5b998f01bb806a69e5 2023-04-03 13:57:35,119 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into 9d0738cb39fc4b5b998f01bb806a69e5(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:35,119 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:35,119 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=13, startTime=1680530254664; duration=0sec 2023-04-03 13:57:35,119 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:35,166 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/1f82a9fbd62247d6a20bfa1c6107de3d 2023-04-03 13:57:35,176 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/451306cfd62b4780bb778418ee97a313 2023-04-03 13:57:35,187 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/2608924e3940483e8f78cdbb02675e87 2023-04-03 13:57:35,190 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/1f82a9fbd62247d6a20bfa1c6107de3d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1f82a9fbd62247d6a20bfa1c6107de3d 2023-04-03 13:57:35,193 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1f82a9fbd62247d6a20bfa1c6107de3d, entries=150, sequenceid=119, filesize=11.6 K 2023-04-03 13:57:35,194 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/451306cfd62b4780bb778418ee97a313 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/451306cfd62b4780bb778418ee97a313 2023-04-03 13:57:35,197 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/451306cfd62b4780bb778418ee97a313, entries=150, sequenceid=119, filesize=11.6 K 2023-04-03 13:57:35,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/2608924e3940483e8f78cdbb02675e87 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2608924e3940483e8f78cdbb02675e87 2023-04-03 13:57:35,201 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2608924e3940483e8f78cdbb02675e87, entries=150, sequenceid=119, filesize=11.6 K 2023-04-03 13:57:35,202 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 653510e400e69700c7ff33f6b46a9396 in 442ms, sequenceid=119, compaction requested=false 2023-04-03 13:57:35,202 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:35,202 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:35,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:35,203 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:35,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:35,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c12abe9[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:35,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,205 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:35,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:35,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:35,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:35,206 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:35,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:35,209 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:35,209 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:35,209 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:35,209 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:35,209 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:35,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,211 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,213 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,215 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,215 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,215 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,215 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,215 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:35,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:35,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:35,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,215 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,216 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:35,216 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:35,216 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,216 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,216 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,216 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:35,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:57:35,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:35,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:35,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:35,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:35,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:35,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:35,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530255289, max timestamp=1680530255301Num uniques 859; 2023-04-03 13:57:35,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530255289, max timestamp=1680530255301Num uniques 859; 2023-04-03 13:57:35,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:35,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530255289, max timestamp=1680530255301Num uniques 859; 2023-04-03 13:57:35,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:35,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:35,316 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:35,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:35,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:35,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:35,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:35,318 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:35,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:35,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:35,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:35,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:35,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:35,321 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:35,321 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,321 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:35,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:35,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:35,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:35,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:35,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:35,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:35,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:35,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:35,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:35,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ffa15f6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:35,339 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:35,339 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:35,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:35,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:35,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:35,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:35,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530315345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,347 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:35,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:35,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:35,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:35,347 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:35,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:35,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,349 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,350 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:35,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:35,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,353 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:35,353 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:35,353 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,353 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,353 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,353 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:35,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530315450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,453 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:35,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:35,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:35,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:35,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:35,457 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:35,457 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:35,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:35,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,458 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,459 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:35,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:35,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:35,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:35,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:35,460 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:35,460 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,460 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:35,460 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:35,460 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:35,460 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,460 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:35,461 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:35,461 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,461 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:35,461 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:35,461 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,461 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:35,461 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:35,462 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:35,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:35,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42de9425[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:35,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:35,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:35,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:35,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:35,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:35,470 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,471 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:35,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:35,473 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:35,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:35,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:35,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:35,473 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:35,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,475 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,476 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:35,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:35,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,478 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:35,478 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:35,478 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:35,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530315485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:35,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530315489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:35,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530315495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:35,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530315495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,578 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:35,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:35,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:35,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:35,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:35,581 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:35,581 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,581 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:35,581 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:35,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,583 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,583 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:35,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:35,583 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:35,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:35,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:35,584 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:35,584 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,584 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:35,590 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,590 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:35,590 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:35,590 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:35,590 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:35,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,595 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,596 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,596 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:35,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:35,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b22ba7b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:35,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,597 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:35,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:35,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:35,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:35,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:35,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,598 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:35,599 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:35,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:35,600 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:35,600 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:35,600 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:35,600 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:35,601 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:35,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,602 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,603 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,607 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:35,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:35,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,607 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:35,607 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:35,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:35,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530315654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/84175e68254b47f4a9effc51790178b4 2023-04-03 13:57:35,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/d66aefa0be8e4e9181670c0496713918 2023-04-03 13:57:35,708 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:35,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:35,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:35,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:35,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:35,714 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:35,714 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,714 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:35,714 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:35,715 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,715 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,716 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,716 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,716 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,716 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:35,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:35,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:35,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:35,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:35,726 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:35,727 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,727 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:35,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:35,748 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:35,752 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:35,752 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:35,752 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:35,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:35,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:35,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:35,767 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,767 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:35,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:35,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52960115[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:35,769 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:35,770 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,770 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,770 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:35,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:35,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:35,771 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,771 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:35,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:35,771 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:35,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:35,775 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:35,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:35,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:35,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:35,775 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:35,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,781 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:35,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,781 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:35,781 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:35,882 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:35,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:35,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:35,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:35,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:35,886 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:35,886 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,886 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:35,886 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:35,886 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,886 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,887 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,887 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:35,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:35,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:35,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:35,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:35,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:35,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:35,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:35,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:35,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:35,888 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,888 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:35,889 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:35,889 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:35,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:35,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:35,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:35,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:35,890 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,890 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,890 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,890 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:35,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8cfb14[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:35,892 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:35,892 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:35,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:35,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:35,893 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,893 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:35,894 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:35,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,894 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,894 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:35,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:35,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:35,901 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:35,901 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:35,901 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:35,901 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:35,901 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:35,904 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,904 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:35,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,904 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:35,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,905 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,906 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:35,908 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:35,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:35,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:35,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:35,908 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:35,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:35,908 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:35,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:35,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530315958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,009 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:36,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:36,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:36,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:36,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:36,029 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:36,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:36,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:36,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:36,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,030 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:36,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:36,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:36,030 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,030 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:36,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:36,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:36,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:36,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:36,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:36,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:36,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:36,045 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,045 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:36,045 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:36,045 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:36,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:36,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,046 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:36,046 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:36,047 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:36,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bc53b3c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:36,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:36,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:36,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:36,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:36,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:36,052 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:36,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:36,056 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:36,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:36,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:36,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:36,056 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:36,062 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,062 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,062 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:36,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:36,082 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:36,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:36,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:36,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:36,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,082 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:36,082 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:36,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:36,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:36,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/3a2803892f314f70a0fe755cd57db69a 2023-04-03 13:57:36,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/84175e68254b47f4a9effc51790178b4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/84175e68254b47f4a9effc51790178b4 2023-04-03 13:57:36,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/84175e68254b47f4a9effc51790178b4, entries=200, sequenceid=131, filesize=14.0 K 2023-04-03 13:57:36,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/d66aefa0be8e4e9181670c0496713918 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d66aefa0be8e4e9181670c0496713918 2023-04-03 13:57:36,183 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:36,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:36,185 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d66aefa0be8e4e9181670c0496713918, entries=150, sequenceid=131, filesize=11.7 K 2023-04-03 13:57:36,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/3a2803892f314f70a0fe755cd57db69a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3a2803892f314f70a0fe755cd57db69a 2023-04-03 13:57:36,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3a2803892f314f70a0fe755cd57db69a, entries=150, sequenceid=131, filesize=11.7 K 2023-04-03 13:57:36,190 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 653510e400e69700c7ff33f6b46a9396 in 926ms, sequenceid=131, compaction requested=true 2023-04-03 13:57:36,190 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:36,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:36,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:36,190 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:36,190 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:36,190 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:36,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:36,191 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38354 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:36,191 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:36,191 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:36,191 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:36,192 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,192 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,192 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/80f90dd1234d42079dd598f44570754a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1f82a9fbd62247d6a20bfa1c6107de3d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/84175e68254b47f4a9effc51790178b4] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=37.5 K 2023-04-03 13:57:36,192 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d23b8cbf94984d5a963b0a9f2720195a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/451306cfd62b4780bb778418ee97a313, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d66aefa0be8e4e9181670c0496713918] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=35.1 K 2023-04-03 13:57:36,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:36,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:36,192 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 80f90dd1234d42079dd598f44570754a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680530253343 2023-04-03 13:57:36,192 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:36,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:36,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:36,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:36,192 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d23b8cbf94984d5a963b0a9f2720195a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680530253343 2023-04-03 13:57:36,193 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,193 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,193 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 451306cfd62b4780bb778418ee97a313, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680530253364 2023-04-03 13:57:36,193 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 1f82a9fbd62247d6a20bfa1c6107de3d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680530253364 2023-04-03 13:57:36,193 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d66aefa0be8e4e9181670c0496713918, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680530255255 2023-04-03 13:57:36,193 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 84175e68254b47f4a9effc51790178b4, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680530255255 2023-04-03 13:57:36,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:36,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:36,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:36,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:36,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:36,196 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:36,196 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:36,196 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:36,197 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:36,197 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:36,197 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,197 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:36,197 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 13:57:36,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:36,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:36,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:36,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:36,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:36,198 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:36,200 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#204 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:36,226 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#205 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:36,282 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/2f48420be9264ce3824e262143345856 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2f48420be9264ce3824e262143345856 2023-04-03 13:57:36,290 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into 2f48420be9264ce3824e262143345856(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:36,290 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:36,290 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=13, startTime=1680530256190; duration=0sec 2023-04-03 13:57:36,290 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:36,290 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:36,291 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35964 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:36,291 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:36,291 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,292 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9d0738cb39fc4b5b998f01bb806a69e5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2608924e3940483e8f78cdbb02675e87, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3a2803892f314f70a0fe755cd57db69a] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=35.1 K 2023-04-03 13:57:36,292 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9d0738cb39fc4b5b998f01bb806a69e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=1680530253343 2023-04-03 13:57:36,292 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2608924e3940483e8f78cdbb02675e87, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680530253364 2023-04-03 13:57:36,293 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3a2803892f314f70a0fe755cd57db69a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680530255255 2023-04-03 13:57:36,305 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:36,344 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f1da8dc8aff34b5db71a078e50d668dc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f1da8dc8aff34b5db71a078e50d668dc 2023-04-03 13:57:36,350 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into f1da8dc8aff34b5db71a078e50d668dc(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:36,350 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:36,350 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=13, startTime=1680530256190; duration=0sec 2023-04-03 13:57:36,350 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:36,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:36,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:36,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:36,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530316488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:36,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44364 deadline: 1680530316591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,658 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/2ff5e5b0065b44afb99c6fe539eb2600 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ff5e5b0065b44afb99c6fe539eb2600 2023-04-03 13:57:36,662 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into 2ff5e5b0065b44afb99c6fe539eb2600(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:36,662 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:36,662 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=13, startTime=1680530256190; duration=0sec 2023-04-03 13:57:36,662 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:36,679 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/8b20d850941a417fb1b1bd5125546383 2023-04-03 13:57:36,688 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/39d6ae9ae8b94b5ca92edd566c4d49cb 2023-04-03 13:57:36,729 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/06dff82d46144139961e067659ec8f3f 2023-04-03 13:57:36,733 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/8b20d850941a417fb1b1bd5125546383 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b20d850941a417fb1b1bd5125546383 2023-04-03 13:57:36,737 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b20d850941a417fb1b1bd5125546383, entries=150, sequenceid=168, filesize=11.8 K 2023-04-03 13:57:36,738 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/39d6ae9ae8b94b5ca92edd566c4d49cb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/39d6ae9ae8b94b5ca92edd566c4d49cb 2023-04-03 13:57:36,746 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/39d6ae9ae8b94b5ca92edd566c4d49cb, entries=150, sequenceid=168, filesize=11.8 K 2023-04-03 13:57:36,747 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/06dff82d46144139961e067659ec8f3f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/06dff82d46144139961e067659ec8f3f 2023-04-03 13:57:36,750 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/06dff82d46144139961e067659ec8f3f, entries=150, sequenceid=168, filesize=11.8 K 2023-04-03 13:57:36,751 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 653510e400e69700c7ff33f6b46a9396 in 554ms, sequenceid=168, compaction requested=false 2023-04-03 13:57:36,751 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:36,751 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:36,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:36,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,754 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:36,754 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:36,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,755 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:36,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@158aad5d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:36,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:36,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:36,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:36,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:36,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:36,758 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:36,758 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:36,758 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:36,758 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:36,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:36,769 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:36,769 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:36,769 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:36,769 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:36,769 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:36,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:36,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,781 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:36,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:36,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:36,785 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:36,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:36,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:36,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:36,786 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:36,786 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:36,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:36,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 13:57:36,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:36,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:36,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:36,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:36,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:36,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:36,820 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/025c0f9db34a4c9b90abbf03552f491a 2023-04-03 13:57:36,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530256835, max timestamp=1680530256846Num uniques 859; 2023-04-03 13:57:36,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530256835, max timestamp=1680530256846Num uniques 859; 2023-04-03 13:57:36,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530256835, max timestamp=1680530256846Num uniques 859; 2023-04-03 13:57:36,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:36,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:36,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:36,889 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:36,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:36,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:36,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:36,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:36,894 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:36,894 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:36,894 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:36,894 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:36,894 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,895 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:36,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:36,896 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,896 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:36,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:36,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:36,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:36,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:36,906 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:36,906 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:36,906 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:36,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/4b55d78b333848bcbd4aa9aefaa6f9cf 2023-04-03 13:57:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:36,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:36,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:36,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:36,919 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:36,919 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,930 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:36,930 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:36,932 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,932 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:36,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f68850[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:36,936 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:36,936 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,936 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,937 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:36,937 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:36,937 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:36,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:36,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:36,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:36,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:36,938 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:36,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:36,941 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:36,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:36,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:36,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:36,941 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:36,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:36,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,946 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:36,954 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:36,954 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:36,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:36,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:36,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:36,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:36,955 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:36,955 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:36,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:36,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:36,956 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:36,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:36,956 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:36,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:36,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530316957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:36,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/5298471369ef49a899045277487bf5da 2023-04-03 13:57:36,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/025c0f9db34a4c9b90abbf03552f491a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/025c0f9db34a4c9b90abbf03552f491a 2023-04-03 13:57:36,984 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/025c0f9db34a4c9b90abbf03552f491a, entries=150, sequenceid=182, filesize=11.8 K 2023-04-03 13:57:36,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/4b55d78b333848bcbd4aa9aefaa6f9cf as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4b55d78b333848bcbd4aa9aefaa6f9cf 2023-04-03 13:57:36,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4b55d78b333848bcbd4aa9aefaa6f9cf, entries=150, sequenceid=182, filesize=11.8 K 2023-04-03 13:57:36,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/5298471369ef49a899045277487bf5da as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5298471369ef49a899045277487bf5da 2023-04-03 13:57:37,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5298471369ef49a899045277487bf5da, entries=150, sequenceid=182, filesize=11.8 K 2023-04-03 13:57:37,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 653510e400e69700c7ff33f6b46a9396 in 221ms, sequenceid=182, compaction requested=true 2023-04-03 13:57:37,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:37,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:37,016 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:37,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:37,016 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:37,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:37,017 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:37,017 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:37,018 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:37,018 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:37,018 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:37,018 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:37,018 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ff5e5b0065b44afb99c6fe539eb2600, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/39d6ae9ae8b94b5ca92edd566c4d49cb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4b55d78b333848bcbd4aa9aefaa6f9cf] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=35.5 K 2023-04-03 13:57:37,018 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2f48420be9264ce3824e262143345856, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b20d850941a417fb1b1bd5125546383, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/025c0f9db34a4c9b90abbf03552f491a] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=35.5 K 2023-04-03 13:57:37,019 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2ff5e5b0065b44afb99c6fe539eb2600, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680530255255 2023-04-03 13:57:37,019 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2f48420be9264ce3824e262143345856, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680530255255 2023-04-03 13:57:37,019 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 39d6ae9ae8b94b5ca92edd566c4d49cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680530255336 2023-04-03 13:57:37,019 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 8b20d850941a417fb1b1bd5125546383, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680530255336 2023-04-03 13:57:37,019 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4b55d78b333848bcbd4aa9aefaa6f9cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680530256483 2023-04-03 13:57:37,020 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 025c0f9db34a4c9b90abbf03552f491a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680530256483 2023-04-03 13:57:37,026 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#213 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:37,039 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#214 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:37,053 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/f44095d976734298af573b350cd9f4eb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f44095d976734298af573b350cd9f4eb 2023-04-03 13:57:37,057 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:37,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:37,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:37,059 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into f44095d976734298af573b350cd9f4eb(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:37,059 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:37,059 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=13, startTime=1680530257016; duration=0sec 2023-04-03 13:57:37,059 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:37,059 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:37,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:37,060 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:37,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:37,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:37,060 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:37,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:37,061 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:37,061 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:37,062 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:37,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:37,062 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:37,062 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:37,062 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f1da8dc8aff34b5db71a078e50d668dc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/06dff82d46144139961e067659ec8f3f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5298471369ef49a899045277487bf5da] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=35.5 K 2023-04-03 13:57:37,062 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,062 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:37,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:37,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:37,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:37,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:37,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:37,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:37,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:37,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:37,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:37,063 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:37,063 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:37,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:37,063 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-03 13:57:37,064 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:37,064 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:37,064 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:37,064 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:37,064 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:37,064 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:37,066 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f1da8dc8aff34b5db71a078e50d668dc, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680530255255 2023-04-03 13:57:37,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:37,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:37,067 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 06dff82d46144139961e067659ec8f3f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680530255336 2023-04-03 13:57:37,067 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5298471369ef49a899045277487bf5da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680530256483 2023-04-03 13:57:37,071 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/55bd1a305ad7429f9f10883fd85b6e17 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/55bd1a305ad7429f9f10883fd85b6e17 2023-04-03 13:57:37,075 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#216 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:37,078 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into 55bd1a305ad7429f9f10883fd85b6e17(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:37,078 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:37,078 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=13, startTime=1680530257016; duration=0sec 2023-04-03 13:57:37,078 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:37,088 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/9db3c57e3ed04dee93e91c20fd9db4d5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9db3c57e3ed04dee93e91c20fd9db4d5 2023-04-03 13:57:37,093 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into 9db3c57e3ed04dee93e91c20fd9db4d5(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:37,093 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:37,094 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=13, startTime=1680530257016; duration=0sec 2023-04-03 13:57:37,094 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:37,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:37,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530317101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:37,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530317204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:37,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530317408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,476 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/974f9b38b13148eaa9563b290f3330ff 2023-04-03 13:57:37,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:37,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530317491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,494 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4123 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:37,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:37,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530317499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,502 DEBUG [Thread-1523] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:37,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:37,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530317500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,504 DEBUG [Thread-1531] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:37,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:37,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530317504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,508 DEBUG [Thread-1527] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:37,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:37,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530317712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,890 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/6b905184d6374f2cb71363a6272bba27 2023-04-03 13:57:37,899 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/2c630cd5276740d7bb46cbfa75a1a83e 2023-04-03 13:57:37,904 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/974f9b38b13148eaa9563b290f3330ff as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/974f9b38b13148eaa9563b290f3330ff 2023-04-03 13:57:37,908 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/974f9b38b13148eaa9563b290f3330ff, entries=150, sequenceid=219, filesize=11.8 K 2023-04-03 13:57:37,908 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/6b905184d6374f2cb71363a6272bba27 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6b905184d6374f2cb71363a6272bba27 2023-04-03 13:57:37,912 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6b905184d6374f2cb71363a6272bba27, entries=150, sequenceid=219, filesize=11.8 K 2023-04-03 13:57:37,913 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/2c630cd5276740d7bb46cbfa75a1a83e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2c630cd5276740d7bb46cbfa75a1a83e 2023-04-03 13:57:37,917 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2c630cd5276740d7bb46cbfa75a1a83e, entries=150, sequenceid=219, filesize=11.8 K 2023-04-03 13:57:37,917 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 653510e400e69700c7ff33f6b46a9396 in 854ms, sequenceid=219, compaction requested=false 2023-04-03 13:57:37,917 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:37,917 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:37,917 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:37,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:37,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:37,919 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:37,919 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:37,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:37,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:37,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:37,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:37,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd889e1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:37,921 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:37,921 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:37,922 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:37,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:37,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:37,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:37,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:37,922 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,922 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:37,923 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:37,923 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:37,923 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,923 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:37,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:37,926 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:37,926 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:37,926 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:37,926 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:37,926 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:37,927 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:37,927 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,928 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:37,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:37,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:37,930 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,942 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:37,942 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:37,942 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:37,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:37,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:37,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:37,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,943 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:37,943 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:37,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:37,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:37,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:37,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:37,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:37,943 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:37,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:37,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,043 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:38,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:38,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:38,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:38,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:38,046 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:38,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:38,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:38,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:38,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:38,048 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,048 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:38,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:38,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:38,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:38,048 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:38,049 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:38,049 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,049 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:38,049 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 13:57:38,050 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:38,050 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:38,050 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:38,051 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:38,051 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:38,051 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:38,072 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/f5685af9da5346368a500b7cc00163f5 2023-04-03 13:57:38,091 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/69c60c15abfa4eb6bda894808deba8c2 2023-04-03 13:57:38,117 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/0b696aa90d954db3a2c28d1b282659f5 2023-04-03 13:57:38,122 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/f5685af9da5346368a500b7cc00163f5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f5685af9da5346368a500b7cc00163f5 2023-04-03 13:57:38,126 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f5685af9da5346368a500b7cc00163f5, entries=150, sequenceid=232, filesize=11.8 K 2023-04-03 13:57:38,133 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/69c60c15abfa4eb6bda894808deba8c2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/69c60c15abfa4eb6bda894808deba8c2 2023-04-03 13:57:38,142 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/69c60c15abfa4eb6bda894808deba8c2, entries=150, sequenceid=232, filesize=11.8 K 2023-04-03 13:57:38,143 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/0b696aa90d954db3a2c28d1b282659f5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0b696aa90d954db3a2c28d1b282659f5 2023-04-03 13:57:38,168 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0b696aa90d954db3a2c28d1b282659f5, entries=150, sequenceid=232, filesize=11.8 K 2023-04-03 13:57:38,169 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 653510e400e69700c7ff33f6b46a9396 in 120ms, sequenceid=232, compaction requested=true 2023-04-03 13:57:38,169 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:38,169 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:38,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:38,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:38,170 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:38,170 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,170 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,171 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,171 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:38,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc64fbd[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:38,173 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:38,173 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,175 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:38,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:38,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:38,175 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,175 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:38,176 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:38,176 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,176 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:38,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:38,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:38,185 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:38,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:38,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:38,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:38,185 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:38,187 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,187 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,188 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,188 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,190 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,192 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,192 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,192 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,192 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,193 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,193 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,193 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,192 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,193 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:38,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:38,193 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,193 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:38,193 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:38,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:38,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:38,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:57:38,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:38,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:38,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:38,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:38,231 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:38,231 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:38,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530258267, max timestamp=1680530258291Num uniques 859; 2023-04-03 13:57:38,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530258267, max timestamp=1680530258291Num uniques 859; 2023-04-03 13:57:38,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530258267, max timestamp=1680530258291Num uniques 859; 2023-04-03 13:57:38,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:38,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:38,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:38,293 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:38,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:38,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:38,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:38,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:38,296 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:38,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:38,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:38,297 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,297 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,297 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,297 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,297 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,298 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:38,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:38,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:38,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:38,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:38,299 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:38,299 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,299 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:38,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:38,302 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:38,302 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:38,302 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:38,302 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:38,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:38,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:38,302 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:38,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,312 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:38,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe6eb5d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:38,314 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:38,314 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,319 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:38,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:38,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:38,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:38,319 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:38,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:38,322 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:38,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:38,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:38,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:38,322 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:38,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,323 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,326 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:38,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:38,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:38,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,327 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:38,327 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:38,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:38,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530318352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,427 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:38,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:38,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:38,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:38,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:38,431 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:38,431 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,431 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:38,431 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:38,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,434 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:38,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:38,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:38,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:38,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:38,438 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,438 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:38,438 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:38,438 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:38,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:38,439 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,439 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,440 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,440 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:38,442 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:38,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c914ef9[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:38,442 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,442 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:38,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:38,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:38,443 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:38,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:38,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,444 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:38,444 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:38,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:38,446 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:38,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:38,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:38,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:38,447 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:38,448 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,448 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,448 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,453 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,453 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,453 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,453 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,453 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:38,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:38,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,453 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:38,453 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:38,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:38,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:38,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530318456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,553 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:38,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:38,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:38,566 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:38,566 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,566 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:38,566 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:38,566 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,566 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,567 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,567 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,568 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,568 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:38,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:38,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:38,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:38,569 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,569 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:38,569 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:38,569 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:38,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:38,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,570 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:38,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cf590b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:38,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:38,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,573 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:38,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:38,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:38,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:38,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:38,575 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:38,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:38,577 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:38,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:38,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:38,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:38,577 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:38,578 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,578 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,578 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,580 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:38,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:38,582 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:38,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,582 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:38,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/4c739e14ffe64162b6b30722a9203de0 2023-04-03 13:57:38,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:38,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530318660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,683 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:38,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:38,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:38,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:38,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:38,702 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:38,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:38,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:38,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/f65fd51d65cd442295b1a6e0d9f0f663 2023-04-03 13:57:38,704 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,704 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,705 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,705 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:38,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:38,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:38,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:38,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:38,714 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:38,714 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,714 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:38,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:38,739 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:38,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:38,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:38,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:38,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:38,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:38,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:38,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78495da8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:38,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:38,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:38,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:38,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:38,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:38,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:38,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:38,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:38,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:38,750 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:38,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:38,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:38,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:38,750 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:38,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:38,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:38,753 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/21d11a4e455b42e48cd753fe36fbe96b 2023-04-03 13:57:38,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:38,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:38,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:38,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:38,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:38,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:38,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:38,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:38,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:38,763 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:38,764 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:38,764 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:38,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/4c739e14ffe64162b6b30722a9203de0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4c739e14ffe64162b6b30722a9203de0 2023-04-03 13:57:38,795 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4c739e14ffe64162b6b30722a9203de0, entries=200, sequenceid=242, filesize=14.1 K 2023-04-03 13:57:38,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/f65fd51d65cd442295b1a6e0d9f0f663 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/f65fd51d65cd442295b1a6e0d9f0f663 2023-04-03 13:57:38,809 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/f65fd51d65cd442295b1a6e0d9f0f663, entries=150, sequenceid=242, filesize=11.8 K 2023-04-03 13:57:38,809 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/21d11a4e455b42e48cd753fe36fbe96b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/21d11a4e455b42e48cd753fe36fbe96b 2023-04-03 13:57:38,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/21d11a4e455b42e48cd753fe36fbe96b, entries=150, sequenceid=242, filesize=11.8 K 2023-04-03 13:57:38,817 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 653510e400e69700c7ff33f6b46a9396 in 586ms, sequenceid=242, compaction requested=true 2023-04-03 13:57:38,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:38,817 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:38,817 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:38,817 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:38,817 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:38,818 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:38,819 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50944 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:38,819 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:38,819 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,819 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f44095d976734298af573b350cd9f4eb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/974f9b38b13148eaa9563b290f3330ff, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f5685af9da5346368a500b7cc00163f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4c739e14ffe64162b6b30722a9203de0] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=49.8 K 2023-04-03 13:57:38,819 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:38,819 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:38,819 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,819 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/55bd1a305ad7429f9f10883fd85b6e17, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6b905184d6374f2cb71363a6272bba27, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/69c60c15abfa4eb6bda894808deba8c2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/f65fd51d65cd442295b1a6e0d9f0f663] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=47.4 K 2023-04-03 13:57:38,819 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f44095d976734298af573b350cd9f4eb, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680530256483 2023-04-03 13:57:38,820 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 55bd1a305ad7429f9f10883fd85b6e17, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680530256483 2023-04-03 13:57:38,820 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 6b905184d6374f2cb71363a6272bba27, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680530256911 2023-04-03 13:57:38,820 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 974f9b38b13148eaa9563b290f3330ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680530256911 2023-04-03 13:57:38,820 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 69c60c15abfa4eb6bda894808deba8c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680530257072 2023-04-03 13:57:38,820 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f5685af9da5346368a500b7cc00163f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680530257072 2023-04-03 13:57:38,820 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f65fd51d65cd442295b1a6e0d9f0f663, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680530258224 2023-04-03 13:57:38,820 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 4c739e14ffe64162b6b30722a9203de0, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680530258218 2023-04-03 13:57:38,838 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#225 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:38,844 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:38,864 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:38,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:38,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:38,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:38,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:38,879 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:38,879 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,879 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:38,879 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:38,880 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,880 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,880 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:38,880 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:38,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:38,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:38,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:38,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:38,905 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/04069646bccc4d6e919175d0a61b9176 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/04069646bccc4d6e919175d0a61b9176 2023-04-03 13:57:38,911 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into 04069646bccc4d6e919175d0a61b9176(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:38,911 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:38,911 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=12, startTime=1680530258817; duration=0sec 2023-04-03 13:57:38,911 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:38,911 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:38,913 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:38,913 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:38,913 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,913 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9db3c57e3ed04dee93e91c20fd9db4d5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2c630cd5276740d7bb46cbfa75a1a83e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0b696aa90d954db3a2c28d1b282659f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/21d11a4e455b42e48cd753fe36fbe96b] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=47.4 K 2023-04-03 13:57:38,913 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9db3c57e3ed04dee93e91c20fd9db4d5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680530256483 2023-04-03 13:57:38,914 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2c630cd5276740d7bb46cbfa75a1a83e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680530256911 2023-04-03 13:57:38,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:38,914 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 0b696aa90d954db3a2c28d1b282659f5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680530257072 2023-04-03 13:57:38,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:38,914 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:38,914 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:38,914 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:38,915 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 21d11a4e455b42e48cd753fe36fbe96b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680530258224 2023-04-03 13:57:38,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:38,918 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:38,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:38,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:38,918 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 13:57:38,918 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:38,919 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:38,919 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:38,919 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:38,919 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:38,919 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:38,923 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:38,944 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/fca40298f287457caa213a9e745c2401 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/fca40298f287457caa213a9e745c2401 2023-04-03 13:57:38,953 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/62cd93050df942d19e931641c8f346e8 2023-04-03 13:57:38,954 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into fca40298f287457caa213a9e745c2401(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:38,954 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:38,954 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=12, startTime=1680530258818; duration=0sec 2023-04-03 13:57:38,954 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:38,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:38,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:38,969 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/51ef557a264a47a18a995116a4833652 2023-04-03 13:57:38,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:38,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530318994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,021 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=280 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/abf9a3f930b947afbfedd550f6c23d3a 2023-04-03 13:57:39,027 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/62cd93050df942d19e931641c8f346e8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/62cd93050df942d19e931641c8f346e8 2023-04-03 13:57:39,035 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/62cd93050df942d19e931641c8f346e8, entries=150, sequenceid=280, filesize=11.9 K 2023-04-03 13:57:39,036 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/51ef557a264a47a18a995116a4833652 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/51ef557a264a47a18a995116a4833652 2023-04-03 13:57:39,054 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/51ef557a264a47a18a995116a4833652, entries=150, sequenceid=280, filesize=11.9 K 2023-04-03 13:57:39,055 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/abf9a3f930b947afbfedd550f6c23d3a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/abf9a3f930b947afbfedd550f6c23d3a 2023-04-03 13:57:39,060 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/abf9a3f930b947afbfedd550f6c23d3a, entries=150, sequenceid=280, filesize=11.9 K 2023-04-03 13:57:39,061 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 653510e400e69700c7ff33f6b46a9396 in 143ms, sequenceid=280, compaction requested=false 2023-04-03 13:57:39,061 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:39,061 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,061 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:39,061 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:39,061 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,061 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:39,061 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:39,062 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,062 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,063 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:39,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfdff[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:39,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:39,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,065 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:39,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:39,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:39,065 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,065 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:39,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:39,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:39,066 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:39,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:39,068 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:39,068 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:39,068 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:39,068 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:39,068 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:39,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,071 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,072 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:39,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:39,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,075 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,075 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:39,075 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:39,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:39,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:39,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 13:57:39,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:39,099 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:39,099 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:39,100 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:39,100 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:39,100 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:39,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530259126, max timestamp=1680530259147Num uniques 859; 2023-04-03 13:57:39,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:39,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530259126, max timestamp=1680530259147Num uniques 859; 2023-04-03 13:57:39,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530259126, max timestamp=1680530259147Num uniques 859; 2023-04-03 13:57:39,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:39,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:39,175 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:39,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:39,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:39,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:39,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:39,177 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:39,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:39,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:39,178 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,178 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,178 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,178 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,179 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,179 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:39,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:39,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:39,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:39,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:39,179 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:39,180 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,180 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:39,182 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:39,182 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:39,182 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:39,182 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:39,182 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:39,183 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,183 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,183 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:39,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66d9d81[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:39,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:39,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,185 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:39,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:39,185 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:39,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:39,186 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:39,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:39,189 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:39,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:39,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:39,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:39,189 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:39,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,191 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,193 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:39,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:39,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,195 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:39,195 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:39,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:39,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530319212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,295 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:39,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:39,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:39,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:39,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:39,304 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:39,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:39,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:39,305 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,305 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,306 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,306 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:39,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:39,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:39,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:39,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:39,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:39,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:39,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:39,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:39,310 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:39,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:39,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,317 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/04b3bed6294f462a962d67420d91a0f1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/04b3bed6294f462a962d67420d91a0f1 2023-04-03 13:57:39,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:39,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:39,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d58b46[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:39,323 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:39,323 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:39,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530319321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:39,325 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:39,325 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:39,325 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:39,325 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into 04b3bed6294f462a962d67420d91a0f1(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:39,325 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:39,326 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=12, startTime=1680530258817; duration=0sec 2023-04-03 13:57:39,326 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:39,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:39,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:39,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:39,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:39,328 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:39,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:39,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:39,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:39,328 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:39,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,332 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,333 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:39,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:39,338 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:39,338 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:39,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,438 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:39,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:39,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:39,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:39,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:39,446 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:39,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:39,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:39,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,448 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,448 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,448 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,448 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:39,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:39,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:39,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:39,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:39,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:39,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:39,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:39,454 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,454 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:39,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:39,454 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:39,454 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:39,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:39,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:39,454 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:39,462 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:39,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bc335c9[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:39,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:39,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:39,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:39,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:39,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:39,470 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,470 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:39,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:39,473 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:39,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:39,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:39,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:39,473 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:39,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,475 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,477 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,478 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:39,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:39,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,478 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,479 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,479 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:39,479 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:39,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/6a8f91e126ad42189b7ea090bbe25aa7 2023-04-03 13:57:39,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:39,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530319526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,579 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:39,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:39,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/493e12a9abaa4e749d46852095c950ed 2023-04-03 13:57:39,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:39,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:39,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:39,594 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:39,594 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,594 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:39,594 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:39,596 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,596 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,596 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,597 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:39,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:39,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:39,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:39,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:39,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:39,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:39,618 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:39,618 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,618 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:39,619 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:39,619 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:39,619 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:39,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:39,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:39,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:39,620 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,620 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,621 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/0ae999cd9d804993a1a9715e48cea057 2023-04-03 13:57:39,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:39,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:39,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@644ec34b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:39,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:39,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,624 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,624 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,624 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:39,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:39,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:39,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:39,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:39,626 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/6a8f91e126ad42189b7ea090bbe25aa7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6a8f91e126ad42189b7ea090bbe25aa7 2023-04-03 13:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:39,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:39,631 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:39,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:39,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:39,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:39,631 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:39,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6a8f91e126ad42189b7ea090bbe25aa7, entries=150, sequenceid=292, filesize=11.9 K 2023-04-03 13:57:39,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/493e12a9abaa4e749d46852095c950ed as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/493e12a9abaa4e749d46852095c950ed 2023-04-03 13:57:39,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:39,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:39,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:39,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:39,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/493e12a9abaa4e749d46852095c950ed, entries=150, sequenceid=292, filesize=11.9 K 2023-04-03 13:57:39,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/0ae999cd9d804993a1a9715e48cea057 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0ae999cd9d804993a1a9715e48cea057 2023-04-03 13:57:39,644 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,644 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:39,644 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,644 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:39,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,644 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:39,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:39,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:39,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:39,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:39,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:39,645 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:39,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,645 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:39,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0ae999cd9d804993a1a9715e48cea057, entries=150, sequenceid=292, filesize=11.9 K 2023-04-03 13:57:39,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 653510e400e69700c7ff33f6b46a9396 in 559ms, sequenceid=292, compaction requested=true 2023-04-03 13:57:39,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:39,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:39,659 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:39,659 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:39,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:39,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:39,662 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:39,663 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:39,663 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,663 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/04069646bccc4d6e919175d0a61b9176, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/62cd93050df942d19e931641c8f346e8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6a8f91e126ad42189b7ea090bbe25aa7] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.1 K 2023-04-03 13:57:39,663 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:39,663 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:39,663 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,664 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/04b3bed6294f462a962d67420d91a0f1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/51ef557a264a47a18a995116a4833652, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/493e12a9abaa4e749d46852095c950ed] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.1 K 2023-04-03 13:57:39,664 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 04069646bccc4d6e919175d0a61b9176, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680530258224 2023-04-03 13:57:39,664 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 04b3bed6294f462a962d67420d91a0f1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680530258224 2023-04-03 13:57:39,664 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 62cd93050df942d19e931641c8f346e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680530258328 2023-04-03 13:57:39,664 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 51ef557a264a47a18a995116a4833652, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680530258328 2023-04-03 13:57:39,665 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 6a8f91e126ad42189b7ea090bbe25aa7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680530258989 2023-04-03 13:57:39,665 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 493e12a9abaa4e749d46852095c950ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680530258989 2023-04-03 13:57:39,689 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#234 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:39,695 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#235 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:39,765 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:39,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:39,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:39,772 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/2e3bb29e79594fe2b6df47a5d8e526a7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2e3bb29e79594fe2b6df47a5d8e526a7 2023-04-03 13:57:39,778 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into 2e3bb29e79594fe2b6df47a5d8e526a7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:39,778 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:39,778 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=13, startTime=1680530259659; duration=0sec 2023-04-03 13:57:39,778 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:39,778 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:39,779 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:39,779 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:39,779 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,780 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/fca40298f287457caa213a9e745c2401, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/abf9a3f930b947afbfedd550f6c23d3a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0ae999cd9d804993a1a9715e48cea057] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.1 K 2023-04-03 13:57:39,780 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fca40298f287457caa213a9e745c2401, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1680530258224 2023-04-03 13:57:39,780 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting abf9a3f930b947afbfedd550f6c23d3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=280, earliestPutTs=1680530258328 2023-04-03 13:57:39,780 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 0ae999cd9d804993a1a9715e48cea057, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680530258989 2023-04-03 13:57:39,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:39,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:39,783 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:39,783 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,783 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:39,783 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:39,784 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,784 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,784 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,784 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:39,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:39,785 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,785 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:39,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:39,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:39,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:39,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:39,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:39,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:39,792 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/4607c60c7f624f6faa7eb6dc97645d79 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4607c60c7f624f6faa7eb6dc97645d79 2023-04-03 13:57:39,793 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#236 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:39,801 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into 4607c60c7f624f6faa7eb6dc97645d79(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:39,801 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:39,801 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=13, startTime=1680530259659; duration=0sec 2023-04-03 13:57:39,801 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:39,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:39,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:39,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:39,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:39,802 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-03 13:57:39,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:39,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:39,803 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:39,803 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:39,803 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:39,803 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:39,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:39,835 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:39,846 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/b82705942c634ac680c2bd7200bf68d0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/b82705942c634ac680c2bd7200bf68d0 2023-04-03 13:57:39,853 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into b82705942c634ac680c2bd7200bf68d0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:39,853 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:39,853 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=13, startTime=1680530259659; duration=0sec 2023-04-03 13:57:39,853 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:39,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:39,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530319869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:39,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:39,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530319974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:40,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530320175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,258 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/8b01748bd20e4dd5928406a1c7ad25bf 2023-04-03 13:57:40,280 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/5a38d3c64ed04dbd830d24a1309f6d3a 2023-04-03 13:57:40,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:40,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530320478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,717 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f7085856a7874ec9b76eb9a64c83f982 2023-04-03 13:57:40,721 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/8b01748bd20e4dd5928406a1c7ad25bf as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b01748bd20e4dd5928406a1c7ad25bf 2023-04-03 13:57:40,724 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b01748bd20e4dd5928406a1c7ad25bf, entries=150, sequenceid=331, filesize=11.9 K 2023-04-03 13:57:40,725 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/5a38d3c64ed04dbd830d24a1309f6d3a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5a38d3c64ed04dbd830d24a1309f6d3a 2023-04-03 13:57:40,728 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5a38d3c64ed04dbd830d24a1309f6d3a, entries=150, sequenceid=331, filesize=11.9 K 2023-04-03 13:57:40,729 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f7085856a7874ec9b76eb9a64c83f982 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f7085856a7874ec9b76eb9a64c83f982 2023-04-03 13:57:40,731 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f7085856a7874ec9b76eb9a64c83f982, entries=150, sequenceid=331, filesize=11.9 K 2023-04-03 13:57:40,732 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 653510e400e69700c7ff33f6b46a9396 in 930ms, sequenceid=331, compaction requested=false 2023-04-03 13:57:40,732 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:40,732 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:40,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:40,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:40,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:40,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:40,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:40,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:40,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:40,734 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:40,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:40,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:40,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e25cbd6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:40,736 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:40,736 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:40,736 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:40,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:40,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:40,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:40,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:40,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:40,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:40,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:40,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:40,737 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:40,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:40,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:40,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:40,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:40,740 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:40,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:40,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:40,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:40,740 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:40,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:40,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:40,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:40,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:40,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:40,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:40,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:40,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:40,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:40,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:40,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:40,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:40,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:40,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:40,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:40,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:40,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:40,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,745 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:40,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:40,745 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:40,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:40,845 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:40,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:40,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:40,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:40,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:40,848 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:40,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:40,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:40,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:40,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:40,849 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:40,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:40,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:40,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:40,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:40,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:40,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:40,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:40,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:40,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:40,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:40,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:40,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:40,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:40,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:40,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:40,858 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:40,858 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:40,858 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 13:57:40,858 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:40,859 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:40,859 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:40,859 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:40,859 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:40,859 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:40,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:40,983 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:41,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530261008, max timestamp=1680530261026Num uniques 859; 2023-04-03 13:57:41,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530261008, max timestamp=1680530261026Num uniques 859; 2023-04-03 13:57:41,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530261008, max timestamp=1680530261026Num uniques 859; 2023-04-03 13:57:41,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:41,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:41,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:41,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:41,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530321078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:41,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:41,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530321182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:41,268 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/3fafa986d7a04ec5898e4b5458fc02da 2023-04-03 13:57:41,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:41,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530321385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:41,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:41,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44348 deadline: 1680530321505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:41,509 DEBUG [Thread-1531] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:41,517 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:41,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44354 deadline: 1680530321516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:41,519 DEBUG [Thread-1525] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:41,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:41,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44328 deadline: 1680530321521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:41,524 DEBUG [Thread-1523] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:41,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:41,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:44344 deadline: 1680530321523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:41,527 DEBUG [Thread-1527] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:57:41,677 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/7d7570d5c9a84106bcc85614084b4846 2023-04-03 13:57:41,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:41,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530321691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,101 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f8d59f4f4a5d445cb21620e2f93ad604 2023-04-03 13:57:42,109 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/3fafa986d7a04ec5898e4b5458fc02da as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3fafa986d7a04ec5898e4b5458fc02da 2023-04-03 13:57:42,113 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3fafa986d7a04ec5898e4b5458fc02da, entries=150, sequenceid=343, filesize=11.9 K 2023-04-03 13:57:42,114 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/7d7570d5c9a84106bcc85614084b4846 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7d7570d5c9a84106bcc85614084b4846 2023-04-03 13:57:42,119 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7d7570d5c9a84106bcc85614084b4846, entries=150, sequenceid=343, filesize=11.9 K 2023-04-03 13:57:42,119 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f8d59f4f4a5d445cb21620e2f93ad604 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8d59f4f4a5d445cb21620e2f93ad604 2023-04-03 13:57:42,123 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8d59f4f4a5d445cb21620e2f93ad604, entries=150, sequenceid=343, filesize=11.9 K 2023-04-03 13:57:42,124 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 653510e400e69700c7ff33f6b46a9396 in 1266ms, sequenceid=343, compaction requested=true 2023-04-03 13:57:42,124 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:42,124 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:42,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:42,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:42,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:42,125 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,125 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,125 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,125 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:42,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b3ffeef[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:42,127 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:42,127 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:42,129 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,129 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:42,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:42,129 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,130 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:42,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:42,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:42,132 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:42,132 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:42,132 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:42,132 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:42,132 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:42,133 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,133 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,133 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,133 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,135 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,136 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,136 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,136 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,136 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,137 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:42,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:42,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,137 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:42,137 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:42,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:42,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:42,202 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 13:57:42,202 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:42,202 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:42,202 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:42,202 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:42,202 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:42,202 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:42,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/362f414fec33476694d1794f61e00996 2023-04-03 13:57:42,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:42,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530322226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,237 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:42,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:42,240 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/768cf378a8c346d78120ea310d3a50e2 2023-04-03 13:57:42,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:42,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:42,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:42,243 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:42,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:42,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:42,244 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,244 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,245 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,245 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,245 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,245 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:42,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:42,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,246 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:42,246 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:42,246 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:42,246 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:42,246 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,246 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:42,248 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:42,248 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,248 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:42,248 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:42,248 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:42,249 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:42,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,250 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:42,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:42,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9c0fd3c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:42,252 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:42,252 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,253 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:42,253 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,253 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:42,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:42,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:42,254 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:42,254 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,254 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:42,254 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:42,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:42,257 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:42,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:42,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:42,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:42,257 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:42,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,258 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,259 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/90f6a60fdd674e8ab26abcfdae08eaca 2023-04-03 13:57:42,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,261 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,262 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,262 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:42,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:42,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,262 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,262 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,263 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:42,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,263 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:42,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:42,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,263 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,264 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/362f414fec33476694d1794f61e00996 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/362f414fec33476694d1794f61e00996 2023-04-03 13:57:42,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/362f414fec33476694d1794f61e00996, entries=200, sequenceid=380, filesize=14.3 K 2023-04-03 13:57:42,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/768cf378a8c346d78120ea310d3a50e2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/768cf378a8c346d78120ea310d3a50e2 2023-04-03 13:57:42,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/768cf378a8c346d78120ea310d3a50e2, entries=150, sequenceid=380, filesize=11.9 K 2023-04-03 13:57:42,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/90f6a60fdd674e8ab26abcfdae08eaca as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/90f6a60fdd674e8ab26abcfdae08eaca 2023-04-03 13:57:42,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/90f6a60fdd674e8ab26abcfdae08eaca, entries=150, sequenceid=380, filesize=11.9 K 2023-04-03 13:57:42,292 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 653510e400e69700c7ff33f6b46a9396 in 90ms, sequenceid=380, compaction requested=true 2023-04-03 13:57:42,292 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:42,292 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:42,292 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:42,292 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:42,293 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:42,293 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:42,294 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,294 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4607c60c7f624f6faa7eb6dc97645d79, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b01748bd20e4dd5928406a1c7ad25bf, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3fafa986d7a04ec5898e4b5458fc02da, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/362f414fec33476694d1794f61e00996] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=50.6 K 2023-04-03 13:57:42,294 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:42,294 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:42,294 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,294 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2e3bb29e79594fe2b6df47a5d8e526a7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5a38d3c64ed04dbd830d24a1309f6d3a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7d7570d5c9a84106bcc85614084b4846, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/768cf378a8c346d78120ea310d3a50e2] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=48.2 K 2023-04-03 13:57:42,294 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 4607c60c7f624f6faa7eb6dc97645d79, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680530258989 2023-04-03 13:57:42,294 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2e3bb29e79594fe2b6df47a5d8e526a7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680530258989 2023-04-03 13:57:42,294 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 8b01748bd20e4dd5928406a1c7ad25bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680530259186 2023-04-03 13:57:42,295 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5a38d3c64ed04dbd830d24a1309f6d3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680530259186 2023-04-03 13:57:42,295 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3fafa986d7a04ec5898e4b5458fc02da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680530259837 2023-04-03 13:57:42,295 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 7d7570d5c9a84106bcc85614084b4846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680530259837 2023-04-03 13:57:42,296 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 362f414fec33476694d1794f61e00996, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530261059 2023-04-03 13:57:42,296 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 768cf378a8c346d78120ea310d3a50e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530261059 2023-04-03 13:57:42,302 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:42,303 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#247 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:42,336 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/ba76fcf4b8b3427b941ea6c8f5009d97 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/ba76fcf4b8b3427b941ea6c8f5009d97 2023-04-03 13:57:42,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 13:57:42,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:42,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:42,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:42,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:42,338 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:42,338 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:42,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:42,344 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into ba76fcf4b8b3427b941ea6c8f5009d97(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:42,344 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:42,344 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=12, startTime=1680530262292; duration=0sec 2023-04-03 13:57:42,344 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:42,344 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:42,345 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:42,346 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:42,346 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,346 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/b82705942c634ac680c2bd7200bf68d0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f7085856a7874ec9b76eb9a64c83f982, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8d59f4f4a5d445cb21620e2f93ad604, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/90f6a60fdd674e8ab26abcfdae08eaca] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=48.2 K 2023-04-03 13:57:42,346 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting b82705942c634ac680c2bd7200bf68d0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680530258989 2023-04-03 13:57:42,346 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f7085856a7874ec9b76eb9a64c83f982, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1680530259186 2023-04-03 13:57:42,347 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f8d59f4f4a5d445cb21620e2f93ad604, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680530259837 2023-04-03 13:57:42,350 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 90f6a60fdd674e8ab26abcfdae08eaca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530261059 2023-04-03 13:57:42,363 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:42,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:42,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:42,385 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#249 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:42,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:42,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:42,387 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:42,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:42,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:42,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,388 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,390 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,390 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:42,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:42,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:42,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:42,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:42,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:42,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:42,412 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:42,412 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:42,412 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,412 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:42,413 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:42,413 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,415 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:42,416 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:42,416 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,416 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:42,416 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:42,416 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/bff2b3eda56a46f786b747cee5062573 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/bff2b3eda56a46f786b747cee5062573 2023-04-03 13:57:42,421 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into bff2b3eda56a46f786b747cee5062573(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:42,421 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:42,421 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=12, startTime=1680530262292; duration=0sec 2023-04-03 13:57:42,421 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:42,427 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,427 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,427 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,427 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:42,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b0633d4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:42,429 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:42,429 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,430 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:42,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:42,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:42,430 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,431 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:42,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530262421, max timestamp=1680530262438Num uniques 859; 2023-04-03 13:57:42,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530262421, max timestamp=1680530262438Num uniques 859; 2023-04-03 13:57:42,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:42,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:42,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530262421, max timestamp=1680530262438Num uniques 859; 2023-04-03 13:57:42,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:42,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:42,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:42,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:42,448 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:42,448 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:42,448 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:42,448 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:42,448 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:42,449 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,449 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,449 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,451 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,452 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,453 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,453 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:42,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:42,453 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,453 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:42,453 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:42,453 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,453 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:42,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:42,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530322488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,553 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:42,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:42,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:42,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:42,562 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:42,562 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,562 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:42,562 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:42,562 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,562 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,563 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,563 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,563 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,563 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:42,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:42,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:42,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:42,565 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:42,565 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,565 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:42,566 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,566 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:42,566 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:42,566 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:42,566 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:42,567 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,567 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:42,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b5d2019[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:42,569 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:42,569 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,569 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:42,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:42,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:42,570 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,570 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:42,570 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:42,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:42,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:42,573 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:42,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:42,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:42,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:42,573 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:42,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,587 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:42,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,587 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,588 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,588 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,588 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,588 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,588 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:42,588 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:42,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:42,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530322592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,688 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:42,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:42,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:42,691 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:42,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:42,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:42,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,693 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,693 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:42,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:42,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:42,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:42,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:42,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:42,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:42,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:42,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:42,694 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,694 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:42,694 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:42,694 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:42,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:42,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,695 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:42,695 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:42,695 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,695 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,696 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,696 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:42,697 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:42,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ff1791b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:42,697 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,698 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:42,698 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,698 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:42,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:42,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:42,698 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,698 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:42,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,699 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,699 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:42,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:42,700 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:42,700 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:42,700 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:42,700 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:42,700 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:42,701 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,701 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,701 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,703 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,705 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:42,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:42,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:42,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,705 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:42,706 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:42,705 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,706 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,727 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/b21606f29c4a4a038a171ffe267a0dda as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/b21606f29c4a4a038a171ffe267a0dda 2023-04-03 13:57:42,731 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into b21606f29c4a4a038a171ffe267a0dda(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:42,731 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:42,731 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=12, startTime=1680530262292; duration=0sec 2023-04-03 13:57:42,731 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:42,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/065eb69a9de341e998f5f99f1c70a491 2023-04-03 13:57:42,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/e4d48ee43b10412e959d9100857157c9 2023-04-03 13:57:42,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:42,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530322794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,806 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:42,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:42,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:42,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:42,809 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:42,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:42,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:42,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,809 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,810 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:42,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:42,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:42,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:42,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:42,814 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,814 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:42,814 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:42,814 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:42,814 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:42,816 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,816 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,816 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,816 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:42,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51d656c4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:42,819 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:42,819 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,819 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,820 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:42,820 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:42,820 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:42,820 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,820 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:42,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,823 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:42,823 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,823 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:42,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:42,824 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:42,824 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:42,824 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:42,824 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:42,824 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:42,826 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,826 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,826 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,826 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,827 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,829 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,829 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,829 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,829 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:42,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:42,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:42,829 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,830 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,830 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:42,830 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,830 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:42,830 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,830 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,830 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,930 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:42,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:42,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:42,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:42,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:42,932 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:42,932 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,932 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:42,932 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:42,933 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,933 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,933 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,934 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,934 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:42,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:42,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:42,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:42,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:42,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:42,934 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:42,935 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,935 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:42,935 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:42,935 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:42,935 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:42,936 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,936 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,936 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,936 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:42,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:42,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7be447b9[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:42,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,938 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:42,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:42,938 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,938 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:42,938 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:42,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:42,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,939 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,939 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:42,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:42,941 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:42,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:42,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:42,941 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:42,941 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:42,942 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,942 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:42,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,942 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:42,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,943 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:42,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,944 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:42,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,945 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:42,945 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:42,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:42,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:42,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:42,945 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,945 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:42,946 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:42,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,946 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:42,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:42,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:42,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:42,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:42,946 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,046 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:43,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:43,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:43,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:43,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:43,050 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:43,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:43,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:43,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:43,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:43,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:43,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:43,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:43,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:43,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:43,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:43,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:43,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:43,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:43,052 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:43,052 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:43,053 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:43,053 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:43,053 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:43,053 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:43,053 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:43,053 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,053 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:43,053 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:43,054 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,054 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:43,054 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,054 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:43,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:43,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:43,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ebfd34[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:43,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:43,056 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,056 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,057 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:43,057 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:43,057 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:43,057 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,057 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:43,057 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:43,057 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,057 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:43,058 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:43,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:43,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:43,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:43,063 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:43,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:43,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:43,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:43,064 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:43,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:43,070 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:43,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,071 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:43,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,072 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,077 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:43,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:43,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:43,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:43,077 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:43,077 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:43,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:43,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:43,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:43,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530323097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,171 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/970d4f67ecd84bef920b5703c67a425e 2023-04-03 13:57:43,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/065eb69a9de341e998f5f99f1c70a491 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/065eb69a9de341e998f5f99f1c70a491 2023-04-03 13:57:43,177 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:43,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:43,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/065eb69a9de341e998f5f99f1c70a491, entries=150, sequenceid=391, filesize=11.9 K 2023-04-03 13:57:43,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/e4d48ee43b10412e959d9100857157c9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e4d48ee43b10412e959d9100857157c9 2023-04-03 13:57:43,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:43,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:43,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:43,193 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:43,193 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:43,193 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:43,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:43,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:43,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:43,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:43,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e4d48ee43b10412e959d9100857157c9, entries=100, sequenceid=391, filesize=9.5 K 2023-04-03 13:57:43,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/970d4f67ecd84bef920b5703c67a425e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/970d4f67ecd84bef920b5703c67a425e 2023-04-03 13:57:43,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:43,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:43,208 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:43,210 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:43,210 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:43,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/970d4f67ecd84bef920b5703c67a425e, entries=100, sequenceid=391, filesize=9.5 K 2023-04-03 13:57:43,212 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:43,212 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:43,212 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:43,212 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:43,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:43,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:43,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:43,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:43,212 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 653510e400e69700c7ff33f6b46a9396 in 875ms, sequenceid=391, compaction requested=false 2023-04-03 13:57:43,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:43,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,213 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:43,213 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,213 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:43,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:43,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:43,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d8660b3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:43,215 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:43,215 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,216 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:43,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:43,216 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,216 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:43,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:43,218 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,218 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:43,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:43,219 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:43,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:43,221 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:43,221 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:43,221 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:43,221 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:43,221 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:43,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:43,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:43,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:43,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,226 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,228 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:43,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,230 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:43,230 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:43,230 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:43,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:43,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:43,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:43,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:43,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:43,330 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:43,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:43,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:43,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:43,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:43,337 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:43,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:43,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:43,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:43,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,338 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:43,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:43,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,339 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:43,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:43,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:43,339 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:43,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:43,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:43,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:43,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:43,340 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:43,340 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:43,340 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-03 13:57:43,340 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:43,340 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:43,340 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:43,340 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:43,341 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:43,341 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:43,360 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/9233ac109cea494a82a3fca1268fbccc 2023-04-03 13:57:43,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:43,603 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:43,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:43,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530323626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:43,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530323729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:43,795 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/8a9a1e68f9f64272a56feb34450c0dce 2023-04-03 13:57:43,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:43,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530323932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,203 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/593f80ec72914d248938155d2b682172 2023-04-03 13:57:44,207 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/9233ac109cea494a82a3fca1268fbccc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/9233ac109cea494a82a3fca1268fbccc 2023-04-03 13:57:44,211 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/9233ac109cea494a82a3fca1268fbccc, entries=150, sequenceid=430, filesize=11.9 K 2023-04-03 13:57:44,213 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/8a9a1e68f9f64272a56feb34450c0dce as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/8a9a1e68f9f64272a56feb34450c0dce 2023-04-03 13:57:44,217 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/8a9a1e68f9f64272a56feb34450c0dce, entries=150, sequenceid=430, filesize=11.9 K 2023-04-03 13:57:44,217 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/593f80ec72914d248938155d2b682172 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/593f80ec72914d248938155d2b682172 2023-04-03 13:57:44,224 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/593f80ec72914d248938155d2b682172, entries=150, sequenceid=430, filesize=11.9 K 2023-04-03 13:57:44,225 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 653510e400e69700c7ff33f6b46a9396 in 885ms, sequenceid=430, compaction requested=true 2023-04-03 13:57:44,225 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:44,225 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:44,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:44,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:44,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:44,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,226 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:44,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:44,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8579b1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:44,229 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:44,229 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,230 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:44,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:44,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:44,230 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,230 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:44,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,231 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:44,231 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,231 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:44,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:44,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:44,236 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:44,236 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:44,236 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:44,236 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:44,236 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:44,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:44,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:57:44,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:44,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:44,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:44,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:44,241 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:44,241 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:44,244 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,244 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,246 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,247 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/48eaeea7ad264cc2b7310581c1f2a1c9 2023-04-03 13:57:44,249 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,254 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,254 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,254 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,254 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,254 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,254 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,254 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,255 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,255 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:44,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,258 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:44,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:44,259 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:44,259 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:44,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:44,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530324304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,359 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:44,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:44,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:44,361 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:44,361 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,361 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:44,361 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:44,362 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,362 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,362 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:44,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:44,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,364 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:44,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:44,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:44,364 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:44,364 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,364 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:44,364 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:44,364 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:44,364 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,364 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:44,364 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:44,364 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,365 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:44,365 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:44,365 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,365 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:44,365 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:44,366 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,366 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,366 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,366 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:44,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:44,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47052c57[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:44,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,368 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:44,368 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:44,368 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:44,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:44,369 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:44,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:44,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:44,372 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:44,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:44,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:44,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:44,372 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:44,373 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,373 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,373 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,373 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,374 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:44,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,376 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,376 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:44,377 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:44,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:44,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530324407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,477 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:44,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:44,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:44,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:44,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:44,479 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:44,479 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,479 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:44,479 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:44,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,480 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,481 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:44,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:44,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:44,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:44,481 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:44,481 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,481 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:44,481 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:44,481 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:44,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:44,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:44,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:44,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:44,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:44,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:44,483 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,483 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,483 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,483 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:44,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:44,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31717bde[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:44,484 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:44,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:44,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:44,485 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,485 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:44,485 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:44,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,485 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:44,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:44,487 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:44,487 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:44,487 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:44,487 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:44,487 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:44,489 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,489 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,489 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,489 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,491 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,491 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,491 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,491 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:44,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:44,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,492 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:44,492 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:44,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:44,592 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:44,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:44,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:44,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:44,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:44,598 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:44,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:44,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:44,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,599 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,599 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,599 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,599 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:44,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:44,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:44,602 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:44,602 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,602 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:44,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:44,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:44,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:44,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:44,603 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:44,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:44,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:44,603 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:44,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,610 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,611 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,611 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:44,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530324610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:44,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fbc06bb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:44,613 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:44,613 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,622 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,622 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:44,622 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:44,622 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:44,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:44,627 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:44,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:44,629 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:44,629 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:44,629 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:44,629 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:44,629 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:44,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:44,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:44,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:44,633 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,633 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,634 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:44,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,634 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:44,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/6c5788c976fb4b7a9e2ded8c7f454ae9 2023-04-03 13:57:44,734 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:44,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:44,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:44,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:44,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:44,737 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:44,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:44,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:44,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:44,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:44,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:44,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:44,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:44,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:44,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:44,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,742 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:44,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac1b00c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:44,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:44,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,744 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:44,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:44,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:44,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:44,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:44,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,745 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:44,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:44,747 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:44,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:44,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:44,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:44,747 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:44,748 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,748 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,748 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,748 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:44,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:44,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,752 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:44,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:44,752 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:44,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,852 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:44,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:44,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:44,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:44,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:44,855 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:44,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:44,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:44,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:44,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:44,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:44,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:44,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:44,857 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:44,857 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:44,858 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:44,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:44,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24c84a97[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:44,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,860 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,860 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:44,860 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:44,860 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:44,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:44,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:44,861 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:44,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:44,863 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:44,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:44,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:44,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:44,863 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:44,864 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,864 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,864 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,865 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,868 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,868 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,868 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,868 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:44,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:44,868 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,868 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,869 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:44,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,869 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:44,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:44,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530324915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,969 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:44,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:44,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:44,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:44,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:44,972 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:44,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:44,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:44,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,973 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:44,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:44,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:44,974 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,974 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:44,974 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:44,974 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:44,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:44,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,975 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,976 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,976 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:44,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1edd6a5b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:44,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:44,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,977 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:44,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:44,977 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:44,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:44,978 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:44,978 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,978 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:44,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:44,980 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:44,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:44,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:44,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:44,980 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:44,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:44,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,981 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:44,982 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:44,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:44,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:44,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:44,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:44,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:44,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:44,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:44,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:44,984 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:44,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:44,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:44,984 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:44,984 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:45,084 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:45,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:45,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/8bca5cb76285465bae8bcd2e645e4592 2023-04-03 13:57:45,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:45,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:45,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:45,088 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:45,088 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:45,088 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:45,088 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:45,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:45,089 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:45,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/48eaeea7ad264cc2b7310581c1f2a1c9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/48eaeea7ad264cc2b7310581c1f2a1c9 2023-04-03 13:57:45,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:45,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:45,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:45,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:45,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:45,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:45,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:45,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:45,094 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:45,094 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:45,094 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:45,094 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:45,094 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/48eaeea7ad264cc2b7310581c1f2a1c9, entries=150, sequenceid=442, filesize=11.9 K 2023-04-03 13:57:45,094 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:45,095 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:45,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:45,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/6c5788c976fb4b7a9e2ded8c7f454ae9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6c5788c976fb4b7a9e2ded8c7f454ae9 2023-04-03 13:57:45,096 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:45,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:45,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:45,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:45,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb2ee5[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:45,097 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:45,097 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:45,098 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:45,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:45,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:45,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:45,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:45,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:45,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,098 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:45,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:45,099 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:45,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:45,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6c5788c976fb4b7a9e2ded8c7f454ae9, entries=100, sequenceid=442, filesize=9.5 K 2023-04-03 13:57:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:45,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/8bca5cb76285465bae8bcd2e645e4592 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8bca5cb76285465bae8bcd2e645e4592 2023-04-03 13:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:45,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:45,101 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:45,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:45,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:45,101 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:45,101 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:45,102 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,102 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:45,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,103 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:45,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:45,103 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:45,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8bca5cb76285465bae8bcd2e645e4592, entries=100, sequenceid=442, filesize=9.5 K 2023-04-03 13:57:45,104 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 653510e400e69700c7ff33f6b46a9396 in 864ms, sequenceid=442, compaction requested=true 2023-04-03 13:57:45,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:45,104 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:45,105 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:45,105 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:45,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:45,105 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:45,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,106 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:45,106 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44640 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:45,106 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:45,106 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:45,106 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:45,106 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:45,106 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/ba76fcf4b8b3427b941ea6c8f5009d97, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e4d48ee43b10412e959d9100857157c9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/8a9a1e68f9f64272a56feb34450c0dce, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6c5788c976fb4b7a9e2ded8c7f454ae9] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=43.6 K 2023-04-03 13:57:45,106 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/b21606f29c4a4a038a171ffe267a0dda, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/065eb69a9de341e998f5f99f1c70a491, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/9233ac109cea494a82a3fca1268fbccc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/48eaeea7ad264cc2b7310581c1f2a1c9] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=48.4 K 2023-04-03 13:57:45,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:45,106 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ba76fcf4b8b3427b941ea6c8f5009d97, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530261059 2023-04-03 13:57:45,106 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting b21606f29c4a4a038a171ffe267a0dda, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530261059 2023-04-03 13:57:45,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,107 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e4d48ee43b10412e959d9100857157c9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680530262225 2023-04-03 13:57:45,107 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 065eb69a9de341e998f5f99f1c70a491, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680530262219 2023-04-03 13:57:45,107 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 8a9a1e68f9f64272a56feb34450c0dce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680530262471 2023-04-03 13:57:45,107 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9233ac109cea494a82a3fca1268fbccc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680530262471 2023-04-03 13:57:45,107 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 6c5788c976fb4b7a9e2ded8c7f454ae9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680530263625 2023-04-03 13:57:45,107 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 48eaeea7ad264cc2b7310581c1f2a1c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680530263611 2023-04-03 13:57:45,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:45,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:45,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:45,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:45,110 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:45,111 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:45,111 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:45,111 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:45,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:45,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:45,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:45,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:45,112 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#258 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:45,113 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#259 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:45,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:45,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:45,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:45,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:45,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,211 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:45,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:45,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:45,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:45,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:45,213 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:45,213 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:45,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:45,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:45,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:45,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,214 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:45,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:45,215 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,215 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:45,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:45,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:45,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:45,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:45,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:45,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:45,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:45,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:45,216 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:45,216 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:45,216 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:45,216 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 13:57:45,216 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:45,217 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:45,217 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:45,217 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:45,217 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:45,217 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:45,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:45,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:45,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:45,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530325452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,521 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/7f61419a75c54fa5a9db86d1953a6d90 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7f61419a75c54fa5a9db86d1953a6d90 2023-04-03 13:57:45,522 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/7eac648da0144ee88de9153e17a120d2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7eac648da0144ee88de9153e17a120d2 2023-04-03 13:57:45,526 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into 7eac648da0144ee88de9153e17a120d2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:45,526 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:45,526 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=12, startTime=1680530265105; duration=0sec 2023-04-03 13:57:45,526 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:45,526 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:45,529 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 44640 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:45,529 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:45,529 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:45,530 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/bff2b3eda56a46f786b747cee5062573, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/970d4f67ecd84bef920b5703c67a425e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/593f80ec72914d248938155d2b682172, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8bca5cb76285465bae8bcd2e645e4592] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=43.6 K 2023-04-03 13:57:45,530 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bff2b3eda56a46f786b747cee5062573, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1680530261059 2023-04-03 13:57:45,530 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 970d4f67ecd84bef920b5703c67a425e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1680530262225 2023-04-03 13:57:45,530 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 593f80ec72914d248938155d2b682172, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680530262471 2023-04-03 13:57:45,531 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 8bca5cb76285465bae8bcd2e645e4592, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680530263625 2023-04-03 13:57:45,531 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into 7f61419a75c54fa5a9db86d1953a6d90(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:45,531 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:45,531 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=12, startTime=1680530265104; duration=0sec 2023-04-03 13:57:45,531 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:45,536 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#261 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:45,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:45,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530325556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,633 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/228cd33350b94610bedf5194c35e7118 2023-04-03 13:57:45,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:45,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530325759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:45,945 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/5ac1233e95f54b2e970523650285a7f4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5ac1233e95f54b2e970523650285a7f4 2023-04-03 13:57:45,950 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into 5ac1233e95f54b2e970523650285a7f4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:45,950 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:45,950 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=12, startTime=1680530265105; duration=0sec 2023-04-03 13:57:45,950 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:46,042 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/2ba270671df0497892e2ca93cc1de2ab 2023-04-03 13:57:46,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:46,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530326061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,453 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/d73324749ef2417c912c68a5adae8920 2023-04-03 13:57:46,458 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/228cd33350b94610bedf5194c35e7118 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/228cd33350b94610bedf5194c35e7118 2023-04-03 13:57:46,462 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/228cd33350b94610bedf5194c35e7118, entries=150, sequenceid=462, filesize=11.9 K 2023-04-03 13:57:46,463 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/2ba270671df0497892e2ca93cc1de2ab as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ba270671df0497892e2ca93cc1de2ab 2023-04-03 13:57:46,467 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ba270671df0497892e2ca93cc1de2ab, entries=150, sequenceid=462, filesize=11.9 K 2023-04-03 13:57:46,468 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/d73324749ef2417c912c68a5adae8920 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/d73324749ef2417c912c68a5adae8920 2023-04-03 13:57:46,472 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/d73324749ef2417c912c68a5adae8920, entries=150, sequenceid=462, filesize=11.9 K 2023-04-03 13:57:46,473 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 653510e400e69700c7ff33f6b46a9396 in 1256ms, sequenceid=462, compaction requested=false 2023-04-03 13:57:46,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:46,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,473 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:46,473 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:46,473 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,473 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:46,473 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:46,475 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,475 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,475 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,475 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:46,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:46,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46cdbf00[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:46,479 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:46,479 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,480 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,480 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:46,480 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:46,480 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:46,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:46,481 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,481 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:46,481 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,481 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:46,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:46,483 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:46,483 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:46,483 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:46,483 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:46,483 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:46,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,489 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,491 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,503 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,503 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,503 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,503 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,503 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,503 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,503 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,503 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,503 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:46,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:46,504 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:46,504 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:46,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:46,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:46,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 13:57:46,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:46,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:46,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:46,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:46,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:46,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:46,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/4e1a7567abf54e5fab855d559f7261fa 2023-04-03 13:57:46,604 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:46,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:46,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:46,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:46,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:46,606 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:46,606 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:46,607 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:46,607 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:46,607 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,607 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,607 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,608 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,608 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,608 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:46,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:46,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:46,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:46,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:46,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:46,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:46,609 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:46,609 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,609 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:46,610 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:46,610 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:46,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:46,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:46,610 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:46,611 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,611 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,611 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,611 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:46,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2037d9bc[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:46,613 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:46,613 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,613 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,614 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:46,614 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:46,614 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:46,614 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,614 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:46,614 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,614 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:46,614 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,614 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:46,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:46,617 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:46,617 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:46,617 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:46,617 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:46,617 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:46,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,619 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,621 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:46,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:46,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,622 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,622 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:46,622 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,622 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:46,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:46,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 425 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530326628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,722 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:46,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:46,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:46,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:46,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:46,724 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:46,725 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:46,725 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:46,725 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:46,725 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,725 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,726 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,726 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,727 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,727 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:46,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:46,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:46,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:46,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:46,727 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:46,727 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:46,727 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:46,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:46,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:46,728 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,728 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:46,728 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:46,728 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:46,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:46,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:46,728 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:46,729 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,729 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,729 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,729 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:46,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55cde053[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:46,731 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:46,731 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,731 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,731 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:46,731 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:46,731 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:46,732 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,732 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:46,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:46,732 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530326731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:46,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:46,734 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:46,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:46,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:46,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:46,734 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:46,735 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,735 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,735 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,736 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:46,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:46,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,739 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:46,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:46,739 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:46,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,839 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:46,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:46,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:46,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:46,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:46,842 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:46,842 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:46,842 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:46,842 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:46,842 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,842 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,843 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,843 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,843 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,843 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:46,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:46,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:46,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:46,844 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,844 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:46,844 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:46,844 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:46,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:46,845 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,845 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,845 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,845 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:46,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d295009[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:46,847 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:46,847 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,848 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,848 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:46,848 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:46,848 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:46,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:46,849 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,849 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:46,849 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,849 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:46,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:46,851 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:46,851 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:46,851 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:46,851 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:46,851 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:46,852 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,852 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,852 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,853 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:46,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,856 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,856 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:46,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:46,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530326935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,957 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:46,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:46,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:46,959 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:46,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:46,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:46,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:46,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,961 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,961 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,962 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,962 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:46,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:46,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:46,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:46,963 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,963 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:46,963 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:46,963 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:46,963 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:46,964 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:46,965 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:46,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2637db13[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:46,965 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,966 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:46,966 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:46,966 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:46,966 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,966 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:46,967 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:46,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:46,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:46,969 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:46,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:46,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:46,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:46,969 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:46,970 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,970 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,970 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:46,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,971 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:46,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:46,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:46,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:46,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:46,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:46,973 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:46,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:46,973 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:46,973 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:46,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:46,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:46,973 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:46,973 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:46,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/1dd44796f0964049ba4c781dedf1cefe 2023-04-03 13:57:47,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/1211b6d564ff4937a3f828a7d3de5250 2023-04-03 13:57:47,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/4e1a7567abf54e5fab855d559f7261fa as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4e1a7567abf54e5fab855d559f7261fa 2023-04-03 13:57:47,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4e1a7567abf54e5fab855d559f7261fa, entries=150, sequenceid=478, filesize=11.9 K 2023-04-03 13:57:47,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/1dd44796f0964049ba4c781dedf1cefe as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1dd44796f0964049ba4c781dedf1cefe 2023-04-03 13:57:47,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1dd44796f0964049ba4c781dedf1cefe, entries=150, sequenceid=478, filesize=11.9 K 2023-04-03 13:57:47,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/1211b6d564ff4937a3f828a7d3de5250 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/1211b6d564ff4937a3f828a7d3de5250 2023-04-03 13:57:47,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/1211b6d564ff4937a3f828a7d3de5250, entries=150, sequenceid=478, filesize=11.9 K 2023-04-03 13:57:47,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 653510e400e69700c7ff33f6b46a9396 in 454ms, sequenceid=478, compaction requested=true 2023-04-03 13:57:47,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:47,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:47,024 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:47,024 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:47,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:47,025 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:47,025 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:47,025 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:47,025 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:47,025 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:47,025 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:47,025 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7f61419a75c54fa5a9db86d1953a6d90, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/228cd33350b94610bedf5194c35e7118, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4e1a7567abf54e5fab855d559f7261fa] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.6 K 2023-04-03 13:57:47,025 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:47,026 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7eac648da0144ee88de9153e17a120d2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ba270671df0497892e2ca93cc1de2ab, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1dd44796f0964049ba4c781dedf1cefe] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.6 K 2023-04-03 13:57:47,026 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 7f61419a75c54fa5a9db86d1953a6d90, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680530262486 2023-04-03 13:57:47,026 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 7eac648da0144ee88de9153e17a120d2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680530262486 2023-04-03 13:57:47,026 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 228cd33350b94610bedf5194c35e7118, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680530264298 2023-04-03 13:57:47,026 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2ba270671df0497892e2ca93cc1de2ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680530264298 2023-04-03 13:57:47,026 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 4e1a7567abf54e5fab855d559f7261fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680530265435 2023-04-03 13:57:47,026 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1dd44796f0964049ba4c781dedf1cefe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680530265435 2023-04-03 13:57:47,049 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#267 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:47,051 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:47,068 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/c96b9c78c3fd4edb959ac4da8af01f5d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c96b9c78c3fd4edb959ac4da8af01f5d 2023-04-03 13:57:47,068 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/e48f6308156342c58fe508809c5ebcb2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e48f6308156342c58fe508809c5ebcb2 2023-04-03 13:57:47,073 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:47,073 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into c96b9c78c3fd4edb959ac4da8af01f5d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:47,073 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into e48f6308156342c58fe508809c5ebcb2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:47,074 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:47,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:47,074 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=13, startTime=1680530267024; duration=0sec 2023-04-03 13:57:47,074 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:47,074 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=13, startTime=1680530267024; duration=0sec 2023-04-03 13:57:47,074 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:47,074 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:47,074 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:47,074 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:47,075 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:47,075 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:47,075 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5ac1233e95f54b2e970523650285a7f4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/d73324749ef2417c912c68a5adae8920, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/1211b6d564ff4937a3f828a7d3de5250] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.6 K 2023-04-03 13:57:47,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:47,075 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5ac1233e95f54b2e970523650285a7f4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680530262486 2023-04-03 13:57:47,075 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d73324749ef2417c912c68a5adae8920, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680530264298 2023-04-03 13:57:47,075 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1211b6d564ff4937a3f828a7d3de5250, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680530265435 2023-04-03 13:57:47,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:47,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:47,076 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:47,076 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:47,076 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:47,076 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:47,076 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:47,076 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:47,077 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:47,077 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:47,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:47,077 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:47,077 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:47,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:47,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:47,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:47,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:47,078 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:47,078 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:47,078 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:47,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:47,093 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:47,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:47,093 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:47,093 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 13:57:47,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:47,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:47,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:47,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:47,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:47,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:47,096 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#269 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:47,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:47,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:47,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:47,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530327264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:47,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:47,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 443 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530327367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:47,512 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/645fdb7b09b5494ea6d834f383de6b1c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/645fdb7b09b5494ea6d834f383de6b1c 2023-04-03 13:57:47,512 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/5ad6a04a3669460d8040d3e964442b26 2023-04-03 13:57:47,517 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into 645fdb7b09b5494ea6d834f383de6b1c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:47,517 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:47,518 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=13, startTime=1680530267025; duration=0sec 2023-04-03 13:57:47,518 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:47,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:47,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 445 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530327569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:47,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:47,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530327872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:47,927 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/216dcd3dc80d4f918a1346209a8b24c8 2023-04-03 13:57:48,348 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=499 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/8e2a6c8099c9459da3dd5453cd98f7ca 2023-04-03 13:57:48,352 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/5ad6a04a3669460d8040d3e964442b26 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/5ad6a04a3669460d8040d3e964442b26 2023-04-03 13:57:48,358 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/5ad6a04a3669460d8040d3e964442b26, entries=150, sequenceid=499, filesize=11.9 K 2023-04-03 13:57:48,358 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/216dcd3dc80d4f918a1346209a8b24c8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/216dcd3dc80d4f918a1346209a8b24c8 2023-04-03 13:57:48,362 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/216dcd3dc80d4f918a1346209a8b24c8, entries=150, sequenceid=499, filesize=11.9 K 2023-04-03 13:57:48,362 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/8e2a6c8099c9459da3dd5453cd98f7ca as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8e2a6c8099c9459da3dd5453cd98f7ca 2023-04-03 13:57:48,366 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8e2a6c8099c9459da3dd5453cd98f7ca, entries=150, sequenceid=499, filesize=11.9 K 2023-04-03 13:57:48,367 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 653510e400e69700c7ff33f6b46a9396 in 1274ms, sequenceid=499, compaction requested=false 2023-04-03 13:57:48,367 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:48,367 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,367 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:48,367 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:48,367 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,367 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:48,367 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:48,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,369 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,370 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:48,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38531c50[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:48,371 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:48,371 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,371 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,372 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:48,372 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:48,372 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:48,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:48,373 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:48,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,373 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:48,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:48,376 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:48,376 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:48,376 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:48,376 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:48,376 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:48,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:48,379 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 13:57:48,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:48,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:48,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:48,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:48,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:48,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:48,388 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,388 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,390 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,392 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:48,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:48,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:48,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,397 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:48,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,397 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:48,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:48,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/42c5d1fa4b5e4c9f9bb42909b8cfb06e 2023-04-03 13:57:48,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:48,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530328435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,497 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:48,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:48,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:48,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:48,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:48,515 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:48,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:48,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:48,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:48,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,517 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,518 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,518 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,518 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:48,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:48,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:48,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:48,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:48,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:48,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:48,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:48,522 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,522 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:48,522 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:48,522 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:48,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:48,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:48,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:48,524 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,524 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,524 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,524 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:48,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a6bc71[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:48,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:48,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,527 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,527 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,527 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:48,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:48,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:48,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:48,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:48,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:48,530 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:48,530 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:48,530 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:48,530 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:48,530 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:48,530 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,531 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,531 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,531 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:48,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,532 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:48,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,534 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,534 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,534 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:48,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:48,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:48,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 467 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530328539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,635 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:48,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:48,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:48,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:48,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:48,637 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:48,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:48,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:48,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:48,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,638 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:48,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:48,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:48,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:48,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:48,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:48,644 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,644 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:48,644 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:48,644 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:48,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:48,652 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,652 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,652 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:48,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:48,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45481d34[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:48,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,656 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,657 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:48,657 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,657 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:48,657 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:48,657 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:48,657 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:48,657 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,657 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,657 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:48,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:48,659 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:48,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:48,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:48,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:48,659 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:48,660 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,660 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,660 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:48,660 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,661 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:48,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:48,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:48,663 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:48,663 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:48,664 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,664 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:48,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:48,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 469 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530328743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,763 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:48,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:48,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:48,766 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:48,766 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:48,766 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:48,766 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:48,767 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,767 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,767 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,767 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,768 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,768 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:48,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:48,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:48,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:48,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:48,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:48,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:48,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:48,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:48,768 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,768 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:48,769 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:48,769 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:48,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:48,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:48,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:48,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,770 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,770 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:48,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d3fa9cd[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:48,771 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:48,771 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,771 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:48,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:48,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:48,772 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,772 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:48,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:48,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:48,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:48,775 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:48,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:48,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:48,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:48,775 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:48,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,778 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:48,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:48,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,780 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:48,780 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:48,782 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,782 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,782 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:48,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:48,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/5765e7ef26b04b839174ba2f66af3e94 2023-04-03 13:57:48,880 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:48,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:48,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:48,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:48,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:48,883 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:48,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:48,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:48,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:48,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,884 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:48,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:48,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:48,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:48,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:48,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:48,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:48,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:48,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:48,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:48,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:48,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:48,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:48,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:48,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:48,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:48,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,888 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:48,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:48,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dae8fa6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:48,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,891 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,891 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,891 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:48,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,891 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:48,891 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:48,891 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:48,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:48,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:48,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:48,897 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:48,897 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:48,897 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:48,897 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:48,897 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:48,905 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,905 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:48,905 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:48,906 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,907 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:48,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:48,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:48,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:48,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:48,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:48,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:48,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:48,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:48,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:48,911 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:48,911 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:48,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:48,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,011 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:49,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:49,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:49,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:49,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:49,014 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:49,014 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,014 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:49,014 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:49,014 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,014 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,015 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,015 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,015 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,015 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:49,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:49,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:49,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,016 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:49,021 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:49,021 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:49,021 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,021 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:49,021 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:49,021 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,021 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:49,021 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:49,021 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,021 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:49,021 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:49,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:49,031 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:49,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5cf156[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:49,031 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:49,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:49,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:49,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:49,035 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,035 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:49,037 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:49,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,037 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:49,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:49,039 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:49,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:49,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:49,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:49,039 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:49,040 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,040 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,040 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,041 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,042 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,043 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,043 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,043 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,043 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,043 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:49,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:49,043 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,043 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,044 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:49,044 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,044 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:49,044 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,044 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:49,044 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:49,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 471 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530329046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,144 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:49,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:49,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:49,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:49,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:49,147 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:49,148 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,148 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:49,148 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:49,148 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,148 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,149 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,149 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:49,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:49,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:49,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:49,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:49,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:49,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:49,159 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,159 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:49,159 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:49,159 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:49,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:49,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:49,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:49,163 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,163 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,164 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:49,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f78f494[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:49,166 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:49,166 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,169 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:49,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:49,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:49,169 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,169 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:49,170 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:49,170 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,170 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:49,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:49,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:49,176 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:49,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:49,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:49,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:49,177 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:49,185 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,185 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,186 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,194 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:49,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:49,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,194 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:49,194 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:49,194 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,195 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:49,256 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/2a4a8a8ff8bd4b27ad70ea93fde0f901 2023-04-03 13:57:49,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/42c5d1fa4b5e4c9f9bb42909b8cfb06e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/42c5d1fa4b5e4c9f9bb42909b8cfb06e 2023-04-03 13:57:49,269 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/42c5d1fa4b5e4c9f9bb42909b8cfb06e, entries=200, sequenceid=514, filesize=14.3 K 2023-04-03 13:57:49,270 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/5765e7ef26b04b839174ba2f66af3e94 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5765e7ef26b04b839174ba2f66af3e94 2023-04-03 13:57:49,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5765e7ef26b04b839174ba2f66af3e94, entries=150, sequenceid=514, filesize=11.9 K 2023-04-03 13:57:49,275 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/2a4a8a8ff8bd4b27ad70ea93fde0f901 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2a4a8a8ff8bd4b27ad70ea93fde0f901 2023-04-03 13:57:49,280 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2a4a8a8ff8bd4b27ad70ea93fde0f901, entries=150, sequenceid=514, filesize=11.9 K 2023-04-03 13:57:49,281 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 653510e400e69700c7ff33f6b46a9396 in 902ms, sequenceid=514, compaction requested=true 2023-04-03 13:57:49,281 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:49,281 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:49,281 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:49,281 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:49,281 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:49,281 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:49,282 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:49,282 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:49,282 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,282 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:49,282 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e48f6308156342c58fe508809c5ebcb2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/5ad6a04a3669460d8040d3e964442b26, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/42c5d1fa4b5e4c9f9bb42909b8cfb06e] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=39.1 K 2023-04-03 13:57:49,282 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:49,282 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,283 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c96b9c78c3fd4edb959ac4da8af01f5d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/216dcd3dc80d4f918a1346209a8b24c8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5765e7ef26b04b839174ba2f66af3e94] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.7 K 2023-04-03 13:57:49,283 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting e48f6308156342c58fe508809c5ebcb2, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680530265435 2023-04-03 13:57:49,283 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c96b9c78c3fd4edb959ac4da8af01f5d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680530265435 2023-04-03 13:57:49,284 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5ad6a04a3669460d8040d3e964442b26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1680530266603 2023-04-03 13:57:49,284 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 216dcd3dc80d4f918a1346209a8b24c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1680530266603 2023-04-03 13:57:49,284 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 42c5d1fa4b5e4c9f9bb42909b8cfb06e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680530267256 2023-04-03 13:57:49,284 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5765e7ef26b04b839174ba2f66af3e94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680530267256 2023-04-03 13:57:49,292 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#276 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:49,295 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:49,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:49,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:49,299 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:49,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:49,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:49,302 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:49,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:49,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:49,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,304 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:49,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:49,310 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:49,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:49,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:49,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:49,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:49,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:49,330 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,330 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:49,331 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 13:57:49,331 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:49,331 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:49,331 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:49,331 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:49,331 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:49,331 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:49,333 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/1d92d11278e14633978dc9450510ef6a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1d92d11278e14633978dc9450510ef6a 2023-04-03 13:57:49,340 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into 1d92d11278e14633978dc9450510ef6a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:49,340 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:49,340 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=13, startTime=1680530269281; duration=0sec 2023-04-03 13:57:49,340 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:49,340 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:49,342 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:49,342 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:49,342 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,342 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/645fdb7b09b5494ea6d834f383de6b1c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8e2a6c8099c9459da3dd5453cd98f7ca, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2a4a8a8ff8bd4b27ad70ea93fde0f901] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.7 K 2023-04-03 13:57:49,342 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 645fdb7b09b5494ea6d834f383de6b1c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680530265435 2023-04-03 13:57:49,343 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 8e2a6c8099c9459da3dd5453cd98f7ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=499, earliestPutTs=1680530266603 2023-04-03 13:57:49,343 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2a4a8a8ff8bd4b27ad70ea93fde0f901, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680530267256 2023-04-03 13:57:49,355 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#279 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:49,357 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/f01c17c7c75045ad9e0201c1b62a6fcd 2023-04-03 13:57:49,399 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/858e5daea9764a64b12f3e5c8e892a1e 2023-04-03 13:57:49,410 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/13851f7e69be44b2a4f6a3d0a9d5d8ec as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/13851f7e69be44b2a4f6a3d0a9d5d8ec 2023-04-03 13:57:49,419 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into 13851f7e69be44b2a4f6a3d0a9d5d8ec(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:49,420 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:49,420 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=13, startTime=1680530269281; duration=0sec 2023-04-03 13:57:49,420 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:49,462 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=532 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/50b03e4b0cbc4b59aea68820da408a74 2023-04-03 13:57:49,468 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/f01c17c7c75045ad9e0201c1b62a6fcd as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f01c17c7c75045ad9e0201c1b62a6fcd 2023-04-03 13:57:49,479 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f01c17c7c75045ad9e0201c1b62a6fcd, entries=150, sequenceid=532, filesize=11.9 K 2023-04-03 13:57:49,486 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/858e5daea9764a64b12f3e5c8e892a1e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/858e5daea9764a64b12f3e5c8e892a1e 2023-04-03 13:57:49,493 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/858e5daea9764a64b12f3e5c8e892a1e, entries=150, sequenceid=532, filesize=11.9 K 2023-04-03 13:57:49,494 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/50b03e4b0cbc4b59aea68820da408a74 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/50b03e4b0cbc4b59aea68820da408a74 2023-04-03 13:57:49,511 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/50b03e4b0cbc4b59aea68820da408a74, entries=150, sequenceid=532, filesize=11.9 K 2023-04-03 13:57:49,512 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 653510e400e69700c7ff33f6b46a9396 in 182ms, sequenceid=532, compaction requested=false 2023-04-03 13:57:49,512 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:49,512 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:49,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:49,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,513 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,513 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,514 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:49,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa94d77[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:49,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:49,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:49,516 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:49,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,517 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:49,517 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:49,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:49,521 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:49,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:49,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:49,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:49,521 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:49,523 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,523 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,523 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,524 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,535 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,538 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,538 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:49,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:49,539 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:49,539 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:49,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:49,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:57:49,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:49,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:49,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:49,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:49,581 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:49,581 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:49,639 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:49,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:49,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:49,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:49,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:49,642 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:49,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530269619, max timestamp=1680530269640Num uniques 859; 2023-04-03 13:57:49,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:49,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:49,642 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:49,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530269619, max timestamp=1680530269640Num uniques 859; 2023-04-03 13:57:49,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:49,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530269619, max timestamp=1680530269640Num uniques 859; 2023-04-03 13:57:49,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:49,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:49,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:49,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:49,658 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:49,658 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,658 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:49,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:49,661 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:49,661 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:49,661 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:49,661 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:49,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:49,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:49,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:49,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:49,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66733c9b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:49,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:49,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,665 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:49,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:49,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,677 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:49,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:49,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:49,677 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:49,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:49,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:49,677 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:49,678 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:49,680 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,680 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,680 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,680 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,681 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,684 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:49,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:49,684 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:49,684 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:49,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:49,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:49,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 514 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530329714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,744 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/dff3d70c535f47d5b2f8d635c4906e1b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/dff3d70c535f47d5b2f8d635c4906e1b 2023-04-03 13:57:49,748 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into dff3d70c535f47d5b2f8d635c4906e1b(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:49,748 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:49,748 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=13, startTime=1680530269281; duration=0sec 2023-04-03 13:57:49,748 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:49,785 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:49,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:49,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:49,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:49,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:49,787 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:49,787 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,787 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:49,787 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:49,788 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,788 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,788 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,788 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,789 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,789 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:49,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:49,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:49,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:49,790 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,790 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. started... 2023-04-03 13:57:49,790 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. as already flushing 2023-04-03 13:57:49,790 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:49,790 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,791 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:49,791 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:49,791 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,791 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,792 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,792 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:49,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b88b742[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:49,794 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:49,794 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,794 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,794 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:49,794 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:49,794 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,794 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:49,794 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:49,795 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,795 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:49,795 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,795 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:49,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:49,797 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:49,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:49,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:49,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:49,798 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:49,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:49,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,799 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:49,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,801 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:49,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:49,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:49,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:49,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:49,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:49,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:49,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:49,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:49,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:49,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:49,804 DEBUG [Thread-1533] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:49,804 DEBUG [Thread-1533] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:49,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:49,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 516 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530329817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:49,890 DEBUG [Thread-1540] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cbc74e7 to 127.0.0.1:57081 2023-04-03 13:57:49,890 DEBUG [Thread-1538] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ade356f to 127.0.0.1:57081 2023-04-03 13:57:49,890 DEBUG [Thread-1538] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:49,890 DEBUG [Thread-1540] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:49,891 DEBUG [Thread-1536] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b7f716c to 127.0.0.1:57081 2023-04-03 13:57:49,891 DEBUG [Thread-1536] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:49,893 DEBUG [Thread-1542] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c5d1710 to 127.0.0.1:57081 2023-04-03 13:57:49,893 DEBUG [Thread-1542] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:49,893 DEBUG [Thread-1534] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e3efec6 to 127.0.0.1:57081 2023-04-03 13:57:49,893 DEBUG [Thread-1534] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:49,904 DEBUG [Thread-1533] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:49,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:50,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/6e9016a825dd41818e11342bda2f91c6 2023-04-03 13:57:50,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:50,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 518 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530330019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:50,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:50,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 520 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530330321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:50,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/31d36922622b4e359184eefbe2e89946 2023-04-03 13:57:50,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:50,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 522 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:44364 deadline: 1680530330825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:50,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/8f60a5687abb42dd8d02b0165c955ae4 2023-04-03 13:57:50,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/6e9016a825dd41818e11342bda2f91c6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6e9016a825dd41818e11342bda2f91c6 2023-04-03 13:57:50,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6e9016a825dd41818e11342bda2f91c6, entries=200, sequenceid=544, filesize=14.3 K 2023-04-03 13:57:50,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/31d36922622b4e359184eefbe2e89946 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/31d36922622b4e359184eefbe2e89946 2023-04-03 13:57:50,842 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/31d36922622b4e359184eefbe2e89946, entries=150, sequenceid=544, filesize=11.9 K 2023-04-03 13:57:50,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/8f60a5687abb42dd8d02b0165c955ae4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8f60a5687abb42dd8d02b0165c955ae4 2023-04-03 13:57:50,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8f60a5687abb42dd8d02b0165c955ae4, entries=150, sequenceid=544, filesize=11.9 K 2023-04-03 13:57:50,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 653510e400e69700c7ff33f6b46a9396 in 1265ms, sequenceid=544, compaction requested=true 2023-04-03 13:57:50,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:50,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:50,847 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:50,847 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:50,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:50,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:50,848 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:50,848 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:50,848 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/A is initiating minor compaction (all files) 2023-04-03 13:57:50,848 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/B is initiating minor compaction (all files) 2023-04-03 13:57:50,848 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/A in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:50,848 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/B in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:50,848 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1d92d11278e14633978dc9450510ef6a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f01c17c7c75045ad9e0201c1b62a6fcd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6e9016a825dd41818e11342bda2f91c6] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=39.2 K 2023-04-03 13:57:50,848 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/dff3d70c535f47d5b2f8d635c4906e1b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/858e5daea9764a64b12f3e5c8e892a1e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/31d36922622b4e359184eefbe2e89946] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.8 K 2023-04-03 13:57:50,848 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 1d92d11278e14633978dc9450510ef6a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680530267256 2023-04-03 13:57:50,848 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting dff3d70c535f47d5b2f8d635c4906e1b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680530267256 2023-04-03 13:57:50,848 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting f01c17c7c75045ad9e0201c1b62a6fcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680530268399 2023-04-03 13:57:50,848 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 858e5daea9764a64b12f3e5c8e892a1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680530268399 2023-04-03 13:57:50,849 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 6e9016a825dd41818e11342bda2f91c6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680530269566 2023-04-03 13:57:50,849 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 31d36922622b4e359184eefbe2e89946, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680530269570 2023-04-03 13:57:50,858 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#B#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:50,858 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#A#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:50,871 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/58a308bd152640ba8ef8a1d3dfb636cb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/58a308bd152640ba8ef8a1d3dfb636cb 2023-04-03 13:57:50,875 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/B of 653510e400e69700c7ff33f6b46a9396 into 58a308bd152640ba8ef8a1d3dfb636cb(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:50,875 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:50,875 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/B, priority=13, startTime=1680530270847; duration=0sec 2023-04-03 13:57:50,875 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:50,875 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:57:50,876 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:57:50,876 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 653510e400e69700c7ff33f6b46a9396/C is initiating minor compaction (all files) 2023-04-03 13:57:50,876 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 653510e400e69700c7ff33f6b46a9396/C in TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:50,876 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/13851f7e69be44b2a4f6a3d0a9d5d8ec, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/50b03e4b0cbc4b59aea68820da408a74, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8f60a5687abb42dd8d02b0165c955ae4] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp, totalSize=36.8 K 2023-04-03 13:57:50,876 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 13851f7e69be44b2a4f6a3d0a9d5d8ec, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1680530267256 2023-04-03 13:57:50,877 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 50b03e4b0cbc4b59aea68820da408a74, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=532, earliestPutTs=1680530268399 2023-04-03 13:57:50,877 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 8f60a5687abb42dd8d02b0165c955ae4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680530269570 2023-04-03 13:57:50,882 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 653510e400e69700c7ff33f6b46a9396#C#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:51,273 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/c9dd34de0af94b53bd2ab7539852285d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/c9dd34de0af94b53bd2ab7539852285d 2023-04-03 13:57:51,295 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/A of 653510e400e69700c7ff33f6b46a9396 into c9dd34de0af94b53bd2ab7539852285d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:51,296 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:51,296 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/A, priority=13, startTime=1680530270847; duration=0sec 2023-04-03 13:57:51,296 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:51,304 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f105db9021ce4b79812dbbaf0abc6592 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f105db9021ce4b79812dbbaf0abc6592 2023-04-03 13:57:51,309 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 653510e400e69700c7ff33f6b46a9396/C of 653510e400e69700c7ff33f6b46a9396 into f105db9021ce4b79812dbbaf0abc6592(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:57:51,309 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:51,309 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396., storeName=653510e400e69700c7ff33f6b46a9396/C, priority=13, startTime=1680530270847; duration=0sec 2023-04-03 13:57:51,309 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:51,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:51,522 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-03 13:57:51,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:51,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:57:51,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:51,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:57:51,523 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:51,523 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:57:51,523 DEBUG [Thread-1531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f837278 to 127.0.0.1:57081 2023-04-03 13:57:51,523 DEBUG [Thread-1531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:51,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/77a0d5d3b3d84af38701ad1c5a8cd78d 2023-04-03 13:57:51,555 DEBUG [Thread-1525] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7117e4df to 127.0.0.1:57081 2023-04-03 13:57:51,555 DEBUG [Thread-1525] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:51,588 DEBUG [Thread-1523] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64e0df67 to 127.0.0.1:57081 2023-04-03 13:57:51,588 DEBUG [Thread-1523] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:51,601 DEBUG [Thread-1527] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c466b6c to 127.0.0.1:57081 2023-04-03 13:57:51,601 DEBUG [Thread-1527] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:51,832 DEBUG [Thread-1529] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46d74611 to 127.0.0.1:57081 2023-04-03 13:57:51,833 DEBUG [Thread-1529] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 387 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 20 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5212 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 4940 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 4909 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5212 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 4950 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 13:57:51,833 INFO [Listener at localhost.localdomain/43477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 13:57:51,833 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29efb2d8 to 127.0.0.1:57081 2023-04-03 13:57:51,833 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:57:51,834 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 13:57:51,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-03 13:57:51,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:51,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 13:57:51,837 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530271837"}]},"ts":"1680530271837"} 2023-04-03 13:57:51,838 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 13:57:51,844 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 13:57:51,845 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=653510e400e69700c7ff33f6b46a9396, UNASSIGN}] 2023-04-03 13:57:51,846 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=653510e400e69700c7ff33f6b46a9396, UNASSIGN 2023-04-03 13:57:51,847 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=653510e400e69700c7ff33f6b46a9396, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:51,847 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530271847"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530271847"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530271847"}]},"ts":"1680530271847"} 2023-04-03 13:57:51,848 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:57:51,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 13:57:51,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/d48f2c1e14894eaf8ffc5c82b3d7ddeb 2023-04-03 13:57:52,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:52,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 653510e400e69700c7ff33f6b46a9396, disabling compactions & flushes 2023-04-03 13:57:52,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 13:57:52,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f13c805af31a452cba67ab2553720c5e 2023-04-03 13:57:52,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/77a0d5d3b3d84af38701ad1c5a8cd78d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/77a0d5d3b3d84af38701ad1c5a8cd78d 2023-04-03 13:57:52,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/77a0d5d3b3d84af38701ad1c5a8cd78d, entries=150, sequenceid=586, filesize=11.9 K 2023-04-03 13:57:52,372 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/d48f2c1e14894eaf8ffc5c82b3d7ddeb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d48f2c1e14894eaf8ffc5c82b3d7ddeb 2023-04-03 13:57:52,376 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d48f2c1e14894eaf8ffc5c82b3d7ddeb, entries=150, sequenceid=586, filesize=11.9 K 2023-04-03 13:57:52,376 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/f13c805af31a452cba67ab2553720c5e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f13c805af31a452cba67ab2553720c5e 2023-04-03 13:57:52,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f13c805af31a452cba67ab2553720c5e, entries=150, sequenceid=586, filesize=11.9 K 2023-04-03 13:57:52,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=26.84 KB/27480 for 653510e400e69700c7ff33f6b46a9396 in 859ms, sequenceid=586, compaction requested=false 2023-04-03 13:57:52,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:52,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. after waiting 0 ms 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:52,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 653510e400e69700c7ff33f6b46a9396 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=A 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=B 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 653510e400e69700c7ff33f6b46a9396, store=C 2023-04-03 13:57:52,381 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:52,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 13:57:52,788 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/7a93786886f24a7d8c344bc9ae8ec0c8 2023-04-03 13:57:52,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 13:57:53,200 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/eca5814ff9d640299b3de75bd08a4658 2023-04-03 13:57:53,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/0d24bf5d36dd4e00b31a8076930a91a6 2023-04-03 13:57:53,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/A/7a93786886f24a7d8c344bc9ae8ec0c8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7a93786886f24a7d8c344bc9ae8ec0c8 2023-04-03 13:57:53,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7a93786886f24a7d8c344bc9ae8ec0c8, entries=150, sequenceid=593, filesize=11.9 K 2023-04-03 13:57:53,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/B/eca5814ff9d640299b3de75bd08a4658 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/eca5814ff9d640299b3de75bd08a4658 2023-04-03 13:57:53,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/eca5814ff9d640299b3de75bd08a4658, entries=150, sequenceid=593, filesize=11.9 K 2023-04-03 13:57:53,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/.tmp/C/0d24bf5d36dd4e00b31a8076930a91a6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0d24bf5d36dd4e00b31a8076930a91a6 2023-04-03 13:57:53,625 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0d24bf5d36dd4e00b31a8076930a91a6, entries=150, sequenceid=593, filesize=11.9 K 2023-04-03 13:57:53,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 653510e400e69700c7ff33f6b46a9396 in 1245ms, sequenceid=593, compaction requested=true 2023-04-03 13:57:53,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3a0571c916a149a4993f6ec0d530c133, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/70f5d613d9414fe198997cf65bfbb8fc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e0538a985c824d0f85935dbabf669296, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2e9d0162ca834c9f8f4cc81750d16366, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/06628fdc841d4761b263ac49b8dc943c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/80f90dd1234d42079dd598f44570754a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3d0cf49404244fb3b940f739234fc620, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1f82a9fbd62247d6a20bfa1c6107de3d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/84175e68254b47f4a9effc51790178b4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2f48420be9264ce3824e262143345856, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b20d850941a417fb1b1bd5125546383, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f44095d976734298af573b350cd9f4eb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/025c0f9db34a4c9b90abbf03552f491a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/974f9b38b13148eaa9563b290f3330ff, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f5685af9da5346368a500b7cc00163f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4c739e14ffe64162b6b30722a9203de0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/04069646bccc4d6e919175d0a61b9176, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/62cd93050df942d19e931641c8f346e8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4607c60c7f624f6faa7eb6dc97645d79, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6a8f91e126ad42189b7ea090bbe25aa7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b01748bd20e4dd5928406a1c7ad25bf, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3fafa986d7a04ec5898e4b5458fc02da, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/362f414fec33476694d1794f61e00996, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/b21606f29c4a4a038a171ffe267a0dda, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/065eb69a9de341e998f5f99f1c70a491, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/9233ac109cea494a82a3fca1268fbccc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7f61419a75c54fa5a9db86d1953a6d90, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/48eaeea7ad264cc2b7310581c1f2a1c9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/228cd33350b94610bedf5194c35e7118, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e48f6308156342c58fe508809c5ebcb2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4e1a7567abf54e5fab855d559f7261fa, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/5ad6a04a3669460d8040d3e964442b26, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/42c5d1fa4b5e4c9f9bb42909b8cfb06e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1d92d11278e14633978dc9450510ef6a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f01c17c7c75045ad9e0201c1b62a6fcd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6e9016a825dd41818e11342bda2f91c6] to archive 2023-04-03 13:57:53,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:57:53,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3a0571c916a149a4993f6ec0d530c133 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3a0571c916a149a4993f6ec0d530c133 2023-04-03 13:57:53,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/70f5d613d9414fe198997cf65bfbb8fc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/70f5d613d9414fe198997cf65bfbb8fc 2023-04-03 13:57:53,648 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e0538a985c824d0f85935dbabf669296 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e0538a985c824d0f85935dbabf669296 2023-04-03 13:57:53,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2e9d0162ca834c9f8f4cc81750d16366 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2e9d0162ca834c9f8f4cc81750d16366 2023-04-03 13:57:53,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/06628fdc841d4761b263ac49b8dc943c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/06628fdc841d4761b263ac49b8dc943c 2023-04-03 13:57:53,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/80f90dd1234d42079dd598f44570754a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/80f90dd1234d42079dd598f44570754a 2023-04-03 13:57:53,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3d0cf49404244fb3b940f739234fc620 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3d0cf49404244fb3b940f739234fc620 2023-04-03 13:57:53,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1f82a9fbd62247d6a20bfa1c6107de3d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1f82a9fbd62247d6a20bfa1c6107de3d 2023-04-03 13:57:53,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/84175e68254b47f4a9effc51790178b4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/84175e68254b47f4a9effc51790178b4 2023-04-03 13:57:53,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2f48420be9264ce3824e262143345856 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/2f48420be9264ce3824e262143345856 2023-04-03 13:57:53,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b20d850941a417fb1b1bd5125546383 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b20d850941a417fb1b1bd5125546383 2023-04-03 13:57:53,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f44095d976734298af573b350cd9f4eb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f44095d976734298af573b350cd9f4eb 2023-04-03 13:57:53,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/025c0f9db34a4c9b90abbf03552f491a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/025c0f9db34a4c9b90abbf03552f491a 2023-04-03 13:57:53,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/974f9b38b13148eaa9563b290f3330ff to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/974f9b38b13148eaa9563b290f3330ff 2023-04-03 13:57:53,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f5685af9da5346368a500b7cc00163f5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f5685af9da5346368a500b7cc00163f5 2023-04-03 13:57:53,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4c739e14ffe64162b6b30722a9203de0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4c739e14ffe64162b6b30722a9203de0 2023-04-03 13:57:53,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/04069646bccc4d6e919175d0a61b9176 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/04069646bccc4d6e919175d0a61b9176 2023-04-03 13:57:53,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/62cd93050df942d19e931641c8f346e8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/62cd93050df942d19e931641c8f346e8 2023-04-03 13:57:53,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4607c60c7f624f6faa7eb6dc97645d79 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4607c60c7f624f6faa7eb6dc97645d79 2023-04-03 13:57:53,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6a8f91e126ad42189b7ea090bbe25aa7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6a8f91e126ad42189b7ea090bbe25aa7 2023-04-03 13:57:53,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b01748bd20e4dd5928406a1c7ad25bf to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/8b01748bd20e4dd5928406a1c7ad25bf 2023-04-03 13:57:53,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3fafa986d7a04ec5898e4b5458fc02da to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/3fafa986d7a04ec5898e4b5458fc02da 2023-04-03 13:57:53,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/362f414fec33476694d1794f61e00996 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/362f414fec33476694d1794f61e00996 2023-04-03 13:57:53,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/b21606f29c4a4a038a171ffe267a0dda to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/b21606f29c4a4a038a171ffe267a0dda 2023-04-03 13:57:53,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/065eb69a9de341e998f5f99f1c70a491 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/065eb69a9de341e998f5f99f1c70a491 2023-04-03 13:57:53,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/9233ac109cea494a82a3fca1268fbccc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/9233ac109cea494a82a3fca1268fbccc 2023-04-03 13:57:53,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7f61419a75c54fa5a9db86d1953a6d90 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7f61419a75c54fa5a9db86d1953a6d90 2023-04-03 13:57:53,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/48eaeea7ad264cc2b7310581c1f2a1c9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/48eaeea7ad264cc2b7310581c1f2a1c9 2023-04-03 13:57:53,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/228cd33350b94610bedf5194c35e7118 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/228cd33350b94610bedf5194c35e7118 2023-04-03 13:57:53,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e48f6308156342c58fe508809c5ebcb2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/e48f6308156342c58fe508809c5ebcb2 2023-04-03 13:57:53,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4e1a7567abf54e5fab855d559f7261fa to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/4e1a7567abf54e5fab855d559f7261fa 2023-04-03 13:57:53,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/5ad6a04a3669460d8040d3e964442b26 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/5ad6a04a3669460d8040d3e964442b26 2023-04-03 13:57:53,673 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/42c5d1fa4b5e4c9f9bb42909b8cfb06e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/42c5d1fa4b5e4c9f9bb42909b8cfb06e 2023-04-03 13:57:53,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1d92d11278e14633978dc9450510ef6a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/1d92d11278e14633978dc9450510ef6a 2023-04-03 13:57:53,674 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f01c17c7c75045ad9e0201c1b62a6fcd to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/f01c17c7c75045ad9e0201c1b62a6fcd 2023-04-03 13:57:53,675 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6e9016a825dd41818e11342bda2f91c6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/6e9016a825dd41818e11342bda2f91c6 2023-04-03 13:57:53,679 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/87998b5f9cd346938e90bb930de989d7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1a315fe584284c64b74ee2cc86e43676, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e01877a66f6c4f1e9f9cc44cd9f7cd15, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4ecd21bff44749caa90e6767abc7239f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c3ec663965fb4100b95eb34273107db3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d23b8cbf94984d5a963b0a9f2720195a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4722ee2ecadb4cd3a09cc9afd9cfe11f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/451306cfd62b4780bb778418ee97a313, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ff5e5b0065b44afb99c6fe539eb2600, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d66aefa0be8e4e9181670c0496713918, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/39d6ae9ae8b94b5ca92edd566c4d49cb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/55bd1a305ad7429f9f10883fd85b6e17, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4b55d78b333848bcbd4aa9aefaa6f9cf, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6b905184d6374f2cb71363a6272bba27, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/69c60c15abfa4eb6bda894808deba8c2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/04b3bed6294f462a962d67420d91a0f1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/f65fd51d65cd442295b1a6e0d9f0f663, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/51ef557a264a47a18a995116a4833652, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2e3bb29e79594fe2b6df47a5d8e526a7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/493e12a9abaa4e749d46852095c950ed, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5a38d3c64ed04dbd830d24a1309f6d3a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7d7570d5c9a84106bcc85614084b4846, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/ba76fcf4b8b3427b941ea6c8f5009d97, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/768cf378a8c346d78120ea310d3a50e2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e4d48ee43b10412e959d9100857157c9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/8a9a1e68f9f64272a56feb34450c0dce, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7eac648da0144ee88de9153e17a120d2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6c5788c976fb4b7a9e2ded8c7f454ae9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ba270671df0497892e2ca93cc1de2ab, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c96b9c78c3fd4edb959ac4da8af01f5d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1dd44796f0964049ba4c781dedf1cefe, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/216dcd3dc80d4f918a1346209a8b24c8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/dff3d70c535f47d5b2f8d635c4906e1b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5765e7ef26b04b839174ba2f66af3e94, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/858e5daea9764a64b12f3e5c8e892a1e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/31d36922622b4e359184eefbe2e89946] to archive 2023-04-03 13:57:53,680 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:57:53,681 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/87998b5f9cd346938e90bb930de989d7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/87998b5f9cd346938e90bb930de989d7 2023-04-03 13:57:53,682 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1a315fe584284c64b74ee2cc86e43676 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1a315fe584284c64b74ee2cc86e43676 2023-04-03 13:57:53,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e01877a66f6c4f1e9f9cc44cd9f7cd15 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e01877a66f6c4f1e9f9cc44cd9f7cd15 2023-04-03 13:57:53,683 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4ecd21bff44749caa90e6767abc7239f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4ecd21bff44749caa90e6767abc7239f 2023-04-03 13:57:53,684 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c3ec663965fb4100b95eb34273107db3 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c3ec663965fb4100b95eb34273107db3 2023-04-03 13:57:53,685 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d23b8cbf94984d5a963b0a9f2720195a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d23b8cbf94984d5a963b0a9f2720195a 2023-04-03 13:57:53,686 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4722ee2ecadb4cd3a09cc9afd9cfe11f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4722ee2ecadb4cd3a09cc9afd9cfe11f 2023-04-03 13:57:53,687 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/451306cfd62b4780bb778418ee97a313 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/451306cfd62b4780bb778418ee97a313 2023-04-03 13:57:53,687 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ff5e5b0065b44afb99c6fe539eb2600 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ff5e5b0065b44afb99c6fe539eb2600 2023-04-03 13:57:53,688 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d66aefa0be8e4e9181670c0496713918 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d66aefa0be8e4e9181670c0496713918 2023-04-03 13:57:53,689 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/39d6ae9ae8b94b5ca92edd566c4d49cb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/39d6ae9ae8b94b5ca92edd566c4d49cb 2023-04-03 13:57:53,690 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/55bd1a305ad7429f9f10883fd85b6e17 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/55bd1a305ad7429f9f10883fd85b6e17 2023-04-03 13:57:53,691 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4b55d78b333848bcbd4aa9aefaa6f9cf to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/4b55d78b333848bcbd4aa9aefaa6f9cf 2023-04-03 13:57:53,691 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6b905184d6374f2cb71363a6272bba27 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6b905184d6374f2cb71363a6272bba27 2023-04-03 13:57:53,692 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/69c60c15abfa4eb6bda894808deba8c2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/69c60c15abfa4eb6bda894808deba8c2 2023-04-03 13:57:53,693 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/04b3bed6294f462a962d67420d91a0f1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/04b3bed6294f462a962d67420d91a0f1 2023-04-03 13:57:53,694 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/f65fd51d65cd442295b1a6e0d9f0f663 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/f65fd51d65cd442295b1a6e0d9f0f663 2023-04-03 13:57:53,695 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/51ef557a264a47a18a995116a4833652 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/51ef557a264a47a18a995116a4833652 2023-04-03 13:57:53,696 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2e3bb29e79594fe2b6df47a5d8e526a7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2e3bb29e79594fe2b6df47a5d8e526a7 2023-04-03 13:57:53,697 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/493e12a9abaa4e749d46852095c950ed to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/493e12a9abaa4e749d46852095c950ed 2023-04-03 13:57:53,698 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5a38d3c64ed04dbd830d24a1309f6d3a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5a38d3c64ed04dbd830d24a1309f6d3a 2023-04-03 13:57:53,699 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7d7570d5c9a84106bcc85614084b4846 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7d7570d5c9a84106bcc85614084b4846 2023-04-03 13:57:53,699 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/ba76fcf4b8b3427b941ea6c8f5009d97 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/ba76fcf4b8b3427b941ea6c8f5009d97 2023-04-03 13:57:53,700 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/768cf378a8c346d78120ea310d3a50e2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/768cf378a8c346d78120ea310d3a50e2 2023-04-03 13:57:53,701 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e4d48ee43b10412e959d9100857157c9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/e4d48ee43b10412e959d9100857157c9 2023-04-03 13:57:53,702 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/8a9a1e68f9f64272a56feb34450c0dce to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/8a9a1e68f9f64272a56feb34450c0dce 2023-04-03 13:57:53,702 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7eac648da0144ee88de9153e17a120d2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/7eac648da0144ee88de9153e17a120d2 2023-04-03 13:57:53,703 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6c5788c976fb4b7a9e2ded8c7f454ae9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/6c5788c976fb4b7a9e2ded8c7f454ae9 2023-04-03 13:57:53,704 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ba270671df0497892e2ca93cc1de2ab to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/2ba270671df0497892e2ca93cc1de2ab 2023-04-03 13:57:53,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c96b9c78c3fd4edb959ac4da8af01f5d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/c96b9c78c3fd4edb959ac4da8af01f5d 2023-04-03 13:57:53,705 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1dd44796f0964049ba4c781dedf1cefe to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/1dd44796f0964049ba4c781dedf1cefe 2023-04-03 13:57:53,706 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/216dcd3dc80d4f918a1346209a8b24c8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/216dcd3dc80d4f918a1346209a8b24c8 2023-04-03 13:57:53,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/dff3d70c535f47d5b2f8d635c4906e1b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/dff3d70c535f47d5b2f8d635c4906e1b 2023-04-03 13:57:53,707 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5765e7ef26b04b839174ba2f66af3e94 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/5765e7ef26b04b839174ba2f66af3e94 2023-04-03 13:57:53,708 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/858e5daea9764a64b12f3e5c8e892a1e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/858e5daea9764a64b12f3e5c8e892a1e 2023-04-03 13:57:53,709 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/31d36922622b4e359184eefbe2e89946 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/31d36922622b4e359184eefbe2e89946 2023-04-03 13:57:53,712 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/a0b2f2e3372741a7aa10ee71e1716592, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5a9ba5181d7c4a3c88805363948e995e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c803b35524194c8cbc9081459f3bfa84, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8aeba027b8745469c7425302deb3989, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3cb44f35f8274d29968a0b27a9c2bdc1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9d0738cb39fc4b5b998f01bb806a69e5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c012c62ad7314abca8678aae48cc4884, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2608924e3940483e8f78cdbb02675e87, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f1da8dc8aff34b5db71a078e50d668dc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3a2803892f314f70a0fe755cd57db69a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/06dff82d46144139961e067659ec8f3f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9db3c57e3ed04dee93e91c20fd9db4d5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5298471369ef49a899045277487bf5da, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2c630cd5276740d7bb46cbfa75a1a83e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0b696aa90d954db3a2c28d1b282659f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/fca40298f287457caa213a9e745c2401, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/21d11a4e455b42e48cd753fe36fbe96b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/abf9a3f930b947afbfedd550f6c23d3a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/b82705942c634ac680c2bd7200bf68d0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0ae999cd9d804993a1a9715e48cea057, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f7085856a7874ec9b76eb9a64c83f982, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8d59f4f4a5d445cb21620e2f93ad604, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/bff2b3eda56a46f786b747cee5062573, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/90f6a60fdd674e8ab26abcfdae08eaca, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/970d4f67ecd84bef920b5703c67a425e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/593f80ec72914d248938155d2b682172, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5ac1233e95f54b2e970523650285a7f4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8bca5cb76285465bae8bcd2e645e4592, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/d73324749ef2417c912c68a5adae8920, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/645fdb7b09b5494ea6d834f383de6b1c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/1211b6d564ff4937a3f828a7d3de5250, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8e2a6c8099c9459da3dd5453cd98f7ca, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/13851f7e69be44b2a4f6a3d0a9d5d8ec, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2a4a8a8ff8bd4b27ad70ea93fde0f901, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/50b03e4b0cbc4b59aea68820da408a74, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8f60a5687abb42dd8d02b0165c955ae4] to archive 2023-04-03 13:57:53,713 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:57:53,714 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/a0b2f2e3372741a7aa10ee71e1716592 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/a0b2f2e3372741a7aa10ee71e1716592 2023-04-03 13:57:53,715 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5a9ba5181d7c4a3c88805363948e995e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5a9ba5181d7c4a3c88805363948e995e 2023-04-03 13:57:53,716 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c803b35524194c8cbc9081459f3bfa84 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c803b35524194c8cbc9081459f3bfa84 2023-04-03 13:57:53,717 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8aeba027b8745469c7425302deb3989 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8aeba027b8745469c7425302deb3989 2023-04-03 13:57:53,717 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3cb44f35f8274d29968a0b27a9c2bdc1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3cb44f35f8274d29968a0b27a9c2bdc1 2023-04-03 13:57:53,718 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9d0738cb39fc4b5b998f01bb806a69e5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9d0738cb39fc4b5b998f01bb806a69e5 2023-04-03 13:57:53,719 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c012c62ad7314abca8678aae48cc4884 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/c012c62ad7314abca8678aae48cc4884 2023-04-03 13:57:53,720 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2608924e3940483e8f78cdbb02675e87 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2608924e3940483e8f78cdbb02675e87 2023-04-03 13:57:53,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f1da8dc8aff34b5db71a078e50d668dc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f1da8dc8aff34b5db71a078e50d668dc 2023-04-03 13:57:53,721 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3a2803892f314f70a0fe755cd57db69a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/3a2803892f314f70a0fe755cd57db69a 2023-04-03 13:57:53,722 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/06dff82d46144139961e067659ec8f3f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/06dff82d46144139961e067659ec8f3f 2023-04-03 13:57:53,723 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9db3c57e3ed04dee93e91c20fd9db4d5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/9db3c57e3ed04dee93e91c20fd9db4d5 2023-04-03 13:57:53,724 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5298471369ef49a899045277487bf5da to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5298471369ef49a899045277487bf5da 2023-04-03 13:57:53,724 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2c630cd5276740d7bb46cbfa75a1a83e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2c630cd5276740d7bb46cbfa75a1a83e 2023-04-03 13:57:53,725 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0b696aa90d954db3a2c28d1b282659f5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0b696aa90d954db3a2c28d1b282659f5 2023-04-03 13:57:53,726 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/fca40298f287457caa213a9e745c2401 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/fca40298f287457caa213a9e745c2401 2023-04-03 13:57:53,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/21d11a4e455b42e48cd753fe36fbe96b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/21d11a4e455b42e48cd753fe36fbe96b 2023-04-03 13:57:53,727 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/abf9a3f930b947afbfedd550f6c23d3a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/abf9a3f930b947afbfedd550f6c23d3a 2023-04-03 13:57:53,728 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/b82705942c634ac680c2bd7200bf68d0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/b82705942c634ac680c2bd7200bf68d0 2023-04-03 13:57:53,729 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0ae999cd9d804993a1a9715e48cea057 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0ae999cd9d804993a1a9715e48cea057 2023-04-03 13:57:53,730 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f7085856a7874ec9b76eb9a64c83f982 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f7085856a7874ec9b76eb9a64c83f982 2023-04-03 13:57:53,731 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8d59f4f4a5d445cb21620e2f93ad604 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f8d59f4f4a5d445cb21620e2f93ad604 2023-04-03 13:57:53,732 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/bff2b3eda56a46f786b747cee5062573 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/bff2b3eda56a46f786b747cee5062573 2023-04-03 13:57:53,732 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/90f6a60fdd674e8ab26abcfdae08eaca to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/90f6a60fdd674e8ab26abcfdae08eaca 2023-04-03 13:57:53,733 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/970d4f67ecd84bef920b5703c67a425e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/970d4f67ecd84bef920b5703c67a425e 2023-04-03 13:57:53,734 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/593f80ec72914d248938155d2b682172 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/593f80ec72914d248938155d2b682172 2023-04-03 13:57:53,735 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5ac1233e95f54b2e970523650285a7f4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/5ac1233e95f54b2e970523650285a7f4 2023-04-03 13:57:53,735 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8bca5cb76285465bae8bcd2e645e4592 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8bca5cb76285465bae8bcd2e645e4592 2023-04-03 13:57:53,736 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/d73324749ef2417c912c68a5adae8920 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/d73324749ef2417c912c68a5adae8920 2023-04-03 13:57:53,737 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/645fdb7b09b5494ea6d834f383de6b1c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/645fdb7b09b5494ea6d834f383de6b1c 2023-04-03 13:57:53,738 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/1211b6d564ff4937a3f828a7d3de5250 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/1211b6d564ff4937a3f828a7d3de5250 2023-04-03 13:57:53,739 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8e2a6c8099c9459da3dd5453cd98f7ca to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8e2a6c8099c9459da3dd5453cd98f7ca 2023-04-03 13:57:53,739 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/13851f7e69be44b2a4f6a3d0a9d5d8ec to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/13851f7e69be44b2a4f6a3d0a9d5d8ec 2023-04-03 13:57:53,740 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2a4a8a8ff8bd4b27ad70ea93fde0f901 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/2a4a8a8ff8bd4b27ad70ea93fde0f901 2023-04-03 13:57:53,741 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/50b03e4b0cbc4b59aea68820da408a74 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/50b03e4b0cbc4b59aea68820da408a74 2023-04-03 13:57:53,742 DEBUG [StoreCloser-TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8f60a5687abb42dd8d02b0165c955ae4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/8f60a5687abb42dd8d02b0165c955ae4 2023-04-03 13:57:53,746 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/recovered.edits/596.seqid, newMaxSeqId=596, maxSeqId=1 2023-04-03 13:57:53,747 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396. 2023-04-03 13:57:53,747 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 653510e400e69700c7ff33f6b46a9396: 2023-04-03 13:57:53,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:53,749 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=653510e400e69700c7ff33f6b46a9396, regionState=CLOSED 2023-04-03 13:57:53,750 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530273749"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530273749"}]},"ts":"1680530273749"} 2023-04-03 13:57:53,751 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-03 13:57:53,752 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 653510e400e69700c7ff33f6b46a9396, server=jenkins-hbase20.apache.org,39695,1680530174856 in 1.9020 sec 2023-04-03 13:57:53,753 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-03 13:57:53,753 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=653510e400e69700c7ff33f6b46a9396, UNASSIGN in 1.9080 sec 2023-04-03 13:57:53,753 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530273753"}]},"ts":"1680530273753"} 2023-04-03 13:57:53,754 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 13:57:53,755 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 13:57:53,756 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9210 sec 2023-04-03 13:57:53,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 13:57:53,943 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-03 13:57:53,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-03 13:57:53,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:53,947 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:53,947 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-03 13:57:53,953 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:53,956 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/recovered.edits] 2023-04-03 13:57:53,961 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/77a0d5d3b3d84af38701ad1c5a8cd78d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/77a0d5d3b3d84af38701ad1c5a8cd78d 2023-04-03 13:57:53,962 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7a93786886f24a7d8c344bc9ae8ec0c8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/7a93786886f24a7d8c344bc9ae8ec0c8 2023-04-03 13:57:53,964 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/c9dd34de0af94b53bd2ab7539852285d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/A/c9dd34de0af94b53bd2ab7539852285d 2023-04-03 13:57:53,966 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/58a308bd152640ba8ef8a1d3dfb636cb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/58a308bd152640ba8ef8a1d3dfb636cb 2023-04-03 13:57:53,967 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d48f2c1e14894eaf8ffc5c82b3d7ddeb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/d48f2c1e14894eaf8ffc5c82b3d7ddeb 2023-04-03 13:57:53,968 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/eca5814ff9d640299b3de75bd08a4658 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/B/eca5814ff9d640299b3de75bd08a4658 2023-04-03 13:57:53,970 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0d24bf5d36dd4e00b31a8076930a91a6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/0d24bf5d36dd4e00b31a8076930a91a6 2023-04-03 13:57:53,971 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f105db9021ce4b79812dbbaf0abc6592 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f105db9021ce4b79812dbbaf0abc6592 2023-04-03 13:57:53,972 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f13c805af31a452cba67ab2553720c5e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/C/f13c805af31a452cba67ab2553720c5e 2023-04-03 13:57:53,973 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/recovered.edits/596.seqid to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396/recovered.edits/596.seqid 2023-04-03 13:57:53,974 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/653510e400e69700c7ff33f6b46a9396 2023-04-03 13:57:53,974 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:57:53,975 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:53,976 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 13:57:53,978 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 13:57:53,979 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:53,979 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 13:57:53,979 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680530273979"}]},"ts":"9223372036854775807"} 2023-04-03 13:57:53,981 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 13:57:53,981 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 653510e400e69700c7ff33f6b46a9396, NAME => 'TestAcidGuarantees,,1680530247678.653510e400e69700c7ff33f6b46a9396.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 13:57:53,981 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 13:57:53,981 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680530273981"}]},"ts":"9223372036854775807"} 2023-04-03 13:57:53,983 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 13:57:53,984 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:53,985 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 41 msec 2023-04-03 13:57:54,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-03 13:57:54,049 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-03 13:57:54,060 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=229 (was 224) - Thread LEAK? -, OpenFileDescriptor=548 (was 509) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=362 (was 350) - SystemLoadAverage LEAK? -, ProcessCount=166 (was 166), AvailableMemoryMB=3852 (was 2835) - AvailableMemoryMB LEAK? - 2023-04-03 13:57:54,070 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=229, OpenFileDescriptor=548, MaxFileDescriptor=60000, SystemLoadAverage=362, ProcessCount=166, AvailableMemoryMB=3852 2023-04-03 13:57:54,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:57:54,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:57:54,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:54,075 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 13:57:54,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-03 13:57:54,076 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 13:57:54,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 13:57:54,077 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,077 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e empty. 2023-04-03 13:57:54,078 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,078 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:57:54,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 13:57:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 13:57:54,489 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:57:54,491 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 899cf3d098e77c9ee3a377e83dee985e, NAME => 'TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp 2023-04-03 13:57:54,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:57:54,506 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 899cf3d098e77c9ee3a377e83dee985e, disabling compactions & flushes 2023-04-03 13:57:54,507 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:54,507 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:54,507 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. after waiting 0 ms 2023-04-03 13:57:54,507 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:54,507 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:54,507 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:57:54,509 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 13:57:54,510 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530274509"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530274509"}]},"ts":"1680530274509"} 2023-04-03 13:57:54,510 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 13:57:54,511 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 13:57:54,511 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530274511"}]},"ts":"1680530274511"} 2023-04-03 13:57:54,512 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 13:57:54,514 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, ASSIGN}] 2023-04-03 13:57:54,515 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, ASSIGN 2023-04-03 13:57:54,516 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=false 2023-04-03 13:57:54,667 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=899cf3d098e77c9ee3a377e83dee985e, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:54,667 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530274667"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530274667"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530274667"}]},"ts":"1680530274667"} 2023-04-03 13:57:54,669 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:57:54,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 13:57:54,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:54,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 899cf3d098e77c9ee3a377e83dee985e, NAME => 'TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:57:54,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:57:54,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,825 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,826 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A 2023-04-03 13:57:54,826 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A 2023-04-03 13:57:54,826 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:54,827 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 899cf3d098e77c9ee3a377e83dee985e columnFamilyName A 2023-04-03 13:57:54,827 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(310): Store=899cf3d098e77c9ee3a377e83dee985e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:54,827 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,828 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B 2023-04-03 13:57:54,829 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B 2023-04-03 13:57:54,829 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:54,829 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 899cf3d098e77c9ee3a377e83dee985e columnFamilyName B 2023-04-03 13:57:54,829 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(310): Store=899cf3d098e77c9ee3a377e83dee985e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:54,830 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,831 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C 2023-04-03 13:57:54,831 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C 2023-04-03 13:57:54,831 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:54,831 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 899cf3d098e77c9ee3a377e83dee985e columnFamilyName C 2023-04-03 13:57:54,832 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(310): Store=899cf3d098e77c9ee3a377e83dee985e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:54,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:54,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:57:54,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:54,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:57:54,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 899cf3d098e77c9ee3a377e83dee985e; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72001497, jitterRate=0.07290591299533844}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:57:54,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:57:54,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., pid=37, masterSystemTime=1680530274820 2023-04-03 13:57:54,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:54,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:54,840 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=899cf3d098e77c9ee3a377e83dee985e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:54,840 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530274840"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530274840"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530274840"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530274840"}]},"ts":"1680530274840"} 2023-04-03 13:57:54,842 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-03 13:57:54,842 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 in 173 msec 2023-04-03 13:57:54,843 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-03 13:57:54,843 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, ASSIGN in 328 msec 2023-04-03 13:57:54,843 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 13:57:54,843 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530274843"}]},"ts":"1680530274843"} 2023-04-03 13:57:54,844 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 13:57:54,845 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 13:57:54,847 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 773 msec 2023-04-03 13:57:55,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 13:57:55,180 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-03 13:57:55,181 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x6fa3f5c5 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:55,192 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7cc9ff5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:55,195 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:55,197 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:55,200 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 13:57:55,202 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:50184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 13:57:55,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:57:55,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:57:55,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 13:57:55,617 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 13:57:55,618 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:57:55,618 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 13:57:55,620 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 13:57:55,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, REOPEN/MOVE}] 2023-04-03 13:57:55,623 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, REOPEN/MOVE 2023-04-03 13:57:55,624 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=899cf3d098e77c9ee3a377e83dee985e, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:55,624 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530275624"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530275624"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530275624"}]},"ts":"1680530275624"} 2023-04-03 13:57:55,625 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:57:55,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:55,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 899cf3d098e77c9ee3a377e83dee985e, disabling compactions & flushes 2023-04-03 13:57:55,778 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:55,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:55,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. after waiting 0 ms 2023-04-03 13:57:55,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:55,781 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 13:57:55,782 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:55,782 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:57:55,782 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 899cf3d098e77c9ee3a377e83dee985e to self. 2023-04-03 13:57:55,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:55,784 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=899cf3d098e77c9ee3a377e83dee985e, regionState=CLOSED 2023-04-03 13:57:55,784 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530275784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530275784"}]},"ts":"1680530275784"} 2023-04-03 13:57:55,786 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-03 13:57:55,786 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 in 160 msec 2023-04-03 13:57:55,786 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=true 2023-04-03 13:57:55,937 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=899cf3d098e77c9ee3a377e83dee985e, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:55,937 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530275937"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530275937"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530275937"}]},"ts":"1680530275937"} 2023-04-03 13:57:55,938 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:57:56,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:56,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 899cf3d098e77c9ee3a377e83dee985e, NAME => 'TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:57:56,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:57:56,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,095 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,096 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A 2023-04-03 13:57:56,096 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A 2023-04-03 13:57:56,096 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:56,097 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 899cf3d098e77c9ee3a377e83dee985e columnFamilyName A 2023-04-03 13:57:56,098 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(310): Store=899cf3d098e77c9ee3a377e83dee985e/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:56,098 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,099 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B 2023-04-03 13:57:56,099 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B 2023-04-03 13:57:56,099 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:56,099 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 899cf3d098e77c9ee3a377e83dee985e columnFamilyName B 2023-04-03 13:57:56,100 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(310): Store=899cf3d098e77c9ee3a377e83dee985e/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:56,100 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,101 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C 2023-04-03 13:57:56,101 DEBUG [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C 2023-04-03 13:57:56,101 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:57:56,101 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 899cf3d098e77c9ee3a377e83dee985e columnFamilyName C 2023-04-03 13:57:56,101 INFO [StoreOpener-899cf3d098e77c9ee3a377e83dee985e-1] regionserver.HStore(310): Store=899cf3d098e77c9ee3a377e83dee985e/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:57:56,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:56,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:57:56,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 899cf3d098e77c9ee3a377e83dee985e; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72747067, jitterRate=0.08401577174663544}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:57:56,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:57:56,109 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., pid=42, masterSystemTime=1680530276090 2023-04-03 13:57:56,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:56,111 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=899cf3d098e77c9ee3a377e83dee985e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:56,111 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530276111"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530276111"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530276111"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530276111"}]},"ts":"1680530276111"} 2023-04-03 13:57:56,113 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-03 13:57:56,113 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 in 174 msec 2023-04-03 13:57:56,114 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-03 13:57:56,114 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, REOPEN/MOVE in 490 msec 2023-04-03 13:57:56,115 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-03 13:57:56,115 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 494 msec 2023-04-03 13:57:56,117 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 911 msec 2023-04-03 13:57:56,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-03 13:57:56,118 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x6ba43635 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,124 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bd9eb59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,124 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x377ba302 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,139 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d243e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,139 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x1eaa86ba to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,147 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74a355de, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,148 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x1d63a8ce to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,156 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57b6628e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,156 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x5d80e5a8 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,164 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5da5030e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,165 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x74d6bf18 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,172 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e7fff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,173 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x060c5cba to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,186 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e30d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,186 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x52249391 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,194 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36a9ece3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,194 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x5ba3ce53 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,205 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ded66ad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,206 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x5144fcff to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:57:56,211 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14d061d2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:57:56,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:56,225 DEBUG [hconnection-0x79ab3b15-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,225 DEBUG [hconnection-0x554d99ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,225 DEBUG [hconnection-0x3f4e2092-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,225 DEBUG [hconnection-0x3900f21a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,226 DEBUG [hconnection-0x30114fbf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,226 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,227 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42066, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:56,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:56,229 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:56,229 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:56,229 DEBUG [hconnection-0x606acb7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,231 DEBUG [hconnection-0x5c6cf8b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,229 DEBUG [hconnection-0x5754b478-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,232 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,229 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:56,232 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:56,233 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42080, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,234 DEBUG [hconnection-0xef2e16d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,234 DEBUG [hconnection-0x1dd634e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:57:56,235 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,235 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,236 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42084, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:56,242 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42118, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:56,244 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:57:56,245 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:56,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:56,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:56,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:56,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:56,256 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:56,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:56,258 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:56,259 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:56,259 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:56,262 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:56,262 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:56,262 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:56,262 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:56,263 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-03 13:57:56,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:57:56,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:56,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:57:56,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:56,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:57:56,266 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:56,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:56,268 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:56,284 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304039464b3b507384b5f969254bc4884adfc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039464b3b507384b5f969254bc4884adfc 2023-04-03 13:57:56,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530276280, max timestamp=1680530276289Num uniques 859; 2023-04-03 13:57:56,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530276280, max timestamp=1680530276289Num uniques 859; 2023-04-03 13:57:56,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:57:56,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530276280, max timestamp=1680530276289Num uniques 859; 2023-04-03 13:57:56,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:57:56,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:57:56,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530336301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530336302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530336300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530336302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680530276299, max timestamp=1680530276300Num uniques 1068; 2023-04-03 13:57:56,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 13:57:56,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680530276299, max timestamp=1680530276300Num uniques 1068; 2023-04-03 13:57:56,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 13:57:56,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530276299, max timestamp=1680530276300Num uniques 1068; 2023-04-03 13:57:56,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=6870, suffix heap size=139040,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=150 2023-04-03 13:57:56,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530336318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530336403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530336403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530336403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530336404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530336419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530336607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530336608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530336608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530336611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530336625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,712 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/e7b27332bd5c4a189b7d57eaa64a2cee 2023-04-03 13:57:56,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530336913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530336914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530336916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530336917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:56,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:56,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530336928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,120 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/1288f737b59644ce849f8b43cd407176 2023-04-03 13:57:57,145 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/fa0f7134f8494d948186c9e3748afc5a 2023-04-03 13:57:57,150 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/e7b27332bd5c4a189b7d57eaa64a2cee as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/e7b27332bd5c4a189b7d57eaa64a2cee 2023-04-03 13:57:57,155 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/e7b27332bd5c4a189b7d57eaa64a2cee, entries=100, sequenceid=13, filesize=18.4 K 2023-04-03 13:57:57,155 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/1288f737b59644ce849f8b43cd407176 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1288f737b59644ce849f8b43cd407176 2023-04-03 13:57:57,162 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1288f737b59644ce849f8b43cd407176, entries=100, sequenceid=13, filesize=9.3 K 2023-04-03 13:57:57,163 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/fa0f7134f8494d948186c9e3748afc5a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fa0f7134f8494d948186c9e3748afc5a 2023-04-03 13:57:57,167 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fa0f7134f8494d948186c9e3748afc5a, entries=100, sequenceid=13, filesize=9.3 K 2023-04-03 13:57:57,168 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=142.33 KB/145742 for 899cf3d098e77c9ee3a377e83dee985e in 905ms, sequenceid=13, compaction requested=false 2023-04-03 13:57:57,168 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:57:57,168 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,168 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:57,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:57,169 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:57,169 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:57,170 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:57,170 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:57,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:57,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43dc26f3[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:57,171 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:57,171 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:57,172 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:57,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:57,173 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:57,173 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:57,173 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:57,173 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,173 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:57,174 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:57,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:57,174 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,175 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:57,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:57,177 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:57,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:57,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:57,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:57,177 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:57,178 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:57,178 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,178 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:57,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,178 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:57,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:57,180 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:57,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:57,182 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:57,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:57,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:57,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:57,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:57,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:57,182 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:57,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:57,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,183 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:57,183 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:57,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:57,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:57,283 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:57,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:57,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:57,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:57,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:57,287 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:57,287 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:57,287 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:57,287 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:57,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:57,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:57,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:57,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:57,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:57,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:57,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:57,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:57,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:57,289 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:57,289 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:57,289 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:57,289 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:57,290 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:57,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:57,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:57,290 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=142.33 KB heapSize=402.63 KB 2023-04-03 13:57:57,290 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,290 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:57,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:57:57,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:57,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:57:57,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:57,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:57:57,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:57,305 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403af8b449f235f4ae8ac3a608f3616fa36 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403af8b449f235f4ae8ac3a608f3616fa36 2023-04-03 13:57:57,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:57,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:57,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530337425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530337426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530337427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530337428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530337437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530337531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530337533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530337535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,712 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=67, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/df86791455bd4d96bc4658e7071c7f51 2023-04-03 13:57:57,725 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/3f72d0231a524062a033e1979b347469 2023-04-03 13:57:57,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530337735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530337736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:57,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:57,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530337741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530338039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530338043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530338044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,134 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=43.93 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/6d5fac33ac494e26bcdc74ed53b967d2 2023-04-03 13:57:58,139 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/df86791455bd4d96bc4658e7071c7f51 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/df86791455bd4d96bc4658e7071c7f51 2023-04-03 13:57:58,142 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/df86791455bd4d96bc4658e7071c7f51, entries=150, sequenceid=67, filesize=25.1 K 2023-04-03 13:57:58,143 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/3f72d0231a524062a033e1979b347469 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3f72d0231a524062a033e1979b347469 2023-04-03 13:57:58,148 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3f72d0231a524062a033e1979b347469, entries=150, sequenceid=67, filesize=11.6 K 2023-04-03 13:57:58,149 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/6d5fac33ac494e26bcdc74ed53b967d2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/6d5fac33ac494e26bcdc74ed53b967d2 2023-04-03 13:57:58,154 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/6d5fac33ac494e26bcdc74ed53b967d2, entries=150, sequenceid=67, filesize=11.6 K 2023-04-03 13:57:58,155 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~142.33 KB/145742, heapSize ~402.58 KB/412240, currentSize=40.25 KB/41220 for 899cf3d098e77c9ee3a377e83dee985e in 865ms, sequenceid=67, compaction requested=false 2023-04-03 13:57:58,155 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:57:58,155 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:58,155 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:58,155 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:58,155 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,156 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:58,157 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:58,157 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:58,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:58,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66bb190a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:58,159 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:58,159 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,160 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:58,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:58,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:58,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:58,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:58,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:58,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:58,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:58,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:58,165 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:58,165 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:58,165 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:58,165 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:58,165 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:58,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:58,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:58,168 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,169 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:58,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:58,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:58,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,177 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,177 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:58,177 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:58,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:58,277 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:58,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:58,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:58,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:58,284 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:58,284 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:58,284 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:58,284 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:58,285 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,285 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:58,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:58,285 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:58,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:58,291 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:58,291 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:58,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:58,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:58,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:58,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:58,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:58,292 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:58,292 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:58,292 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 13:57:58,292 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:57:58,292 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:58,292 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:57:58,292 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:58,292 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:57:58,292 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:57:58,316 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403e84ecdda92304e4d8477b14017ed624c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e84ecdda92304e4d8477b14017ed624c 2023-04-03 13:57:58,332 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=76, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/95d04ded359847d6a10578a7baeca19b 2023-04-03 13:57:58,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:58,434 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:58,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530278487, max timestamp=1680530278496Num uniques 859; 2023-04-03 13:57:58,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530278487, max timestamp=1680530278496Num uniques 859; 2023-04-03 13:57:58,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:58,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:58,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530278487, max timestamp=1680530278496Num uniques 859; 2023-04-03 13:57:58,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:57:58,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530278526, max timestamp=1680530278553Num uniques 1018; 2023-04-03 13:57:58,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 13:57:58,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530278526, max timestamp=1680530278553Num uniques 1018; 2023-04-03 13:57:58,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530278526, max timestamp=1680530278553Num uniques 1018; 2023-04-03 13:57:58,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 13:57:58,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 13:57:58,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530338576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530338581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530278571, max timestamp=1680530278585Num uniques 1077; 2023-04-03 13:57:58,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530278571, max timestamp=1680530278585Num uniques 1077; 2023-04-03 13:57:58,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-03 13:57:58,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-03 13:57:58,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530278571, max timestamp=1680530278585Num uniques 1027; 2023-04-03 13:57:58,588 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-03 13:57:58,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530338627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530338628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530338629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530338687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530338688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530338734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530338734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530338739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,760 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/fa57f645fd7d4dcb9ec8d212ca518210 2023-04-03 13:57:58,784 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/e5782ab1a0d4407da6b958b34738462b 2023-04-03 13:57:58,792 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/95d04ded359847d6a10578a7baeca19b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/95d04ded359847d6a10578a7baeca19b 2023-04-03 13:57:58,800 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/95d04ded359847d6a10578a7baeca19b, entries=150, sequenceid=76, filesize=25.1 K 2023-04-03 13:57:58,801 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/fa57f645fd7d4dcb9ec8d212ca518210 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/fa57f645fd7d4dcb9ec8d212ca518210 2023-04-03 13:57:58,814 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/fa57f645fd7d4dcb9ec8d212ca518210, entries=150, sequenceid=76, filesize=11.6 K 2023-04-03 13:57:58,815 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/e5782ab1a0d4407da6b958b34738462b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/e5782ab1a0d4407da6b958b34738462b 2023-04-03 13:57:58,818 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/e5782ab1a0d4407da6b958b34738462b, entries=150, sequenceid=76, filesize=11.6 K 2023-04-03 13:57:58,818 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 899cf3d098e77c9ee3a377e83dee985e in 526ms, sequenceid=76, compaction requested=true 2023-04-03 13:57:58,819 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:57:58,819 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:58,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:58,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:58,819 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,827 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:58,827 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:58,828 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,828 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:58,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,829 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:58,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:58,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195376a2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:58,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:58,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,833 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:58,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:58,833 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:58,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:58,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:58,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:58,835 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:58,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:58,836 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:58,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:58,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:58,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:58,836 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:58,837 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,837 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:58,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:58,837 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,837 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:58,840 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,846 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:58,846 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:58,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:58,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:58,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:58,847 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:58,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,847 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:58,847 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:58,847 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:58,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:58,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:57:58,900 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=147.60 KB heapSize=413.77 KB 2023-04-03 13:57:58,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:57:58,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:58,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:57:58,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:58,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:57:58,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:57:58,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530338917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530338921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,942 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040359a39df43d9f4efc85126e8bc72b0128 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040359a39df43d9f4efc85126e8bc72b0128 2023-04-03 13:57:58,947 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:58,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:58,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:58,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530338942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530338943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,952 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:58,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530338949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:58,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:58,958 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:58,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:58,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:58,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:58,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:58,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:58,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:58,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:58,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:58,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:58,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:58,961 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:58,961 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:58,961 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:58,961 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:58,961 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:58,963 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:58,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,964 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:58,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:58,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17eff06d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:58,965 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:58,965 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,966 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,966 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:58,966 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:58,966 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:58,966 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,966 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:58,967 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:58,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:58,967 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:58,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:58,970 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:58,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:58,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:58,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:58,970 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:58,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:58,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:58,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:58,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,972 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:58,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,975 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:58,975 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:58,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:58,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:58,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:58,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:58,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:58,975 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:58,976 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:58,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:58,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:58,976 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:58,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:58,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:58,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:58,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530339024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530339025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,076 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:59,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:59,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:59,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:59,079 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:59,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:59,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:59,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,080 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,081 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:59,081 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:59,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:59,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:59,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:59,081 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:59,082 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:59,082 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,082 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:59,082 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:59,082 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,082 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:59,082 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:59,082 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,082 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:59,082 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:59,083 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,083 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:59,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35a6a7dd[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:59,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:59,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,087 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:59,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:59,087 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,087 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:59,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:59,088 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:59,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:59,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:59,090 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:59,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:59,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:59,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:59,091 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:59,091 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,091 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,092 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,094 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:59,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:59,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,095 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:59,095 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:59,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:59,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,195 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:59,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:59,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:59,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:59,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:59,198 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:59,198 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,198 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:59,198 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:59,198 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,198 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,199 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,199 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,199 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,199 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:59,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:59,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:59,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:59,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:59,200 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:59,200 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,200 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:59,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:59,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:59,201 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,201 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:59,201 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:59,201 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:59,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:59,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:59,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:59,202 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:59,203 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:59,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@beb8b2a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:59,203 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,203 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:59,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:59,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:59,204 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,204 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:59,204 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:59,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:59,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:59,206 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:59,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:59,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:59,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:59,206 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:59,207 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,207 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,207 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,207 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,208 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,209 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,209 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,209 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:59,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,210 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:59,210 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:59,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530339229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530339233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530339260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530339260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530339261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,277 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 13:57:59,310 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:59,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:59,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:59,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:59,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:59,317 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:59,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:59,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:59,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:59,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:59,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:59,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:59,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:59,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:59,327 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,327 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:59,327 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:59,343 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,343 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,343 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:59,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:59,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@90f1a93[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:59,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:59,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:59,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:59,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:59,357 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,357 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/d8ba66011f5947fc94d1229f8f801265 2023-04-03 13:57:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:59,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:59,372 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:59,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:59,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:59,372 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:59,372 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:59,373 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,374 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,374 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,374 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,375 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,386 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:59,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:59,386 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,387 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:59,387 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:59,387 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,387 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:59,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/3add8ba5b55f4e1887440d616e344ec6 2023-04-03 13:57:59,487 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:59,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:59,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:59,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:59,490 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:59,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:59,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:59,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,490 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,492 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,492 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:59,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:59,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:59,494 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,494 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:59,494 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:59,494 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:59,494 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:59,496 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,497 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,497 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,497 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:59,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@610bff47[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:59,498 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:59,498 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,498 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:59,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:59,499 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:59,499 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:59,499 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,499 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:59,500 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,500 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:59,500 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:59,500 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:59,500 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:59,500 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:59,500 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:59,500 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:59,501 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,501 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,501 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:59,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:59,504 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,504 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,504 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530339535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530339538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,605 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:59,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:59,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:59,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:59,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:59,610 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:59,610 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,610 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:59,610 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:59,611 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,611 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,611 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,611 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,612 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,612 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:59,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:59,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:59,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:59,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,626 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:59,636 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:59,637 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,637 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:59,637 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:59,637 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:59,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,638 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:59,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611334f4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:59,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:59,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,644 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:59,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:59,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:59,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:59,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,646 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:59,646 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,646 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:59,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:59,652 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:59,652 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:59,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:59,653 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:59,653 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:59,662 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,667 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,669 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,671 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:59,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:59,672 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:59,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:57:59,672 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:59,772 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:59,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:59,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:59,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:59,776 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:59,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:59,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:59,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530339770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530339771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,777 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:57:59,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530339772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:59,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:59,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:59,794 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:59,794 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,794 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:57:59,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:57:59,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:57:59,798 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,798 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:57:59,798 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:57:59,798 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:57:59,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:57:59,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:57:59,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:57:59,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:57:59,801 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:57:59,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36765127[Count = 0] remaining members to acquire global barrier 2023-04-03 13:57:59,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:57:59,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,804 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:57:59,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:57:59,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:57:59,804 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,804 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:57:59,804 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:57:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:57:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:57:59,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:57:59,807 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:57:59,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:57:59,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:57:59,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:57:59,807 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:57:59,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:57:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:57:59,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:57:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:57:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:57:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:57:59,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,813 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:57:59,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,821 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:57:59,822 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:57:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:57:59,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:57:59,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:57:59,822 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:57:59,822 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:57:59,822 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:57:59,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/5b68078a5c434d5f83b748be28480e3d 2023-04-03 13:57:59,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/d8ba66011f5947fc94d1229f8f801265 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/d8ba66011f5947fc94d1229f8f801265 2023-04-03 13:57:59,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/d8ba66011f5947fc94d1229f8f801265, entries=200, sequenceid=147, filesize=32.1 K 2023-04-03 13:57:59,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/3add8ba5b55f4e1887440d616e344ec6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3add8ba5b55f4e1887440d616e344ec6 2023-04-03 13:57:59,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3add8ba5b55f4e1887440d616e344ec6, entries=150, sequenceid=147, filesize=11.8 K 2023-04-03 13:57:59,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/5b68078a5c434d5f83b748be28480e3d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5b68078a5c434d5f83b748be28480e3d 2023-04-03 13:57:59,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5b68078a5c434d5f83b748be28480e3d, entries=150, sequenceid=147, filesize=11.8 K 2023-04-03 13:57:59,894 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~413.72 KB/423648, currentSize=33.54 KB/34350 for 899cf3d098e77c9ee3a377e83dee985e in 994ms, sequenceid=147, compaction requested=true 2023-04-03 13:57:59,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:57:59,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:59,894 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:59,894 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:57:59,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:57:59,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:57:59,896 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:59,896 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103176 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:57:59,896 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/B is initiating minor compaction (all files) 2023-04-03 13:57:59,896 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/A is initiating minor compaction (all files) 2023-04-03 13:57:59,896 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/B in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,896 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1288f737b59644ce849f8b43cd407176, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3f72d0231a524062a033e1979b347469, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/fa57f645fd7d4dcb9ec8d212ca518210, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3add8ba5b55f4e1887440d616e344ec6] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=44.3 K 2023-04-03 13:57:59,896 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/A in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:57:59,896 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/e7b27332bd5c4a189b7d57eaa64a2cee, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/df86791455bd4d96bc4658e7071c7f51, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/95d04ded359847d6a10578a7baeca19b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/d8ba66011f5947fc94d1229f8f801265] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=100.8 K 2023-04-03 13:57:59,896 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1288f737b59644ce849f8b43cd407176, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680530276244 2023-04-03 13:57:59,897 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting e7b27332bd5c4a189b7d57eaa64a2cee, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680530276244 2023-04-03 13:57:59,898 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 3f72d0231a524062a033e1979b347469, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680530276308 2023-04-03 13:57:59,899 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting df86791455bd4d96bc4658e7071c7f51, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680530276308 2023-04-03 13:57:59,899 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fa57f645fd7d4dcb9ec8d212ca518210, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1680530277424 2023-04-03 13:57:59,899 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 95d04ded359847d6a10578a7baeca19b, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1680530277424 2023-04-03 13:57:59,899 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 3add8ba5b55f4e1887440d616e344ec6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680530278624 2023-04-03 13:57:59,899 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting d8ba66011f5947fc94d1229f8f801265, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680530278624 2023-04-03 13:57:59,922 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:57:59,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:57:59,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:57:59,957 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#B#compaction#307 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:57:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:57:59,958 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:57:59,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:57:59,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:57:59,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,959 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,960 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:57:59,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:57:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:57:59,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:57:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:57:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:57:59,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:57:59,962 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:57:59,962 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:57:59,979 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#A#compaction#306 average throughput is 0.32 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:57:59,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:57:59,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:57:59,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:57:59,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:57:59,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:00,006 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:00,006 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:00,006 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:00,006 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:00,006 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 13:58:00,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:00,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:00,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:00,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:00,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:00,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:00,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:00,053 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:00,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530280115, max timestamp=1680530280153Num uniques 859; 2023-04-03 13:58:00,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:00,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530280115, max timestamp=1680530280153Num uniques 859; 2023-04-03 13:58:00,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:00,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530280115, max timestamp=1680530280153Num uniques 859; 2023-04-03 13:58:00,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:00,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530280236, max timestamp=1680530280281Num uniques 1068; 2023-04-03 13:58:00,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 13:58:00,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530280236, max timestamp=1680530280281Num uniques 1068; 2023-04-03 13:58:00,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530280236, max timestamp=1680530280281Num uniques 1068; 2023-04-03 13:58:00,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 13:58:00,287 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 13:58:00,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530280344, max timestamp=1680530280364Num uniques 1077; 2023-04-03 13:58:00,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:00,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530280344, max timestamp=1680530280364Num uniques 1077; 2023-04-03 13:58:00,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:00,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530280344, max timestamp=1680530280364Num uniques 1077; 2023-04-03 13:58:00,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:00,411 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/ea371f743761460fbaccdecc690265d7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/ea371f743761460fbaccdecc690265d7 2023-04-03 13:58:00,414 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/B of 899cf3d098e77c9ee3a377e83dee985e into ea371f743761460fbaccdecc690265d7(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:00,415 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:00,415 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/B, priority=12, startTime=1680530279894; duration=0sec 2023-04-03 13:58:00,415 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:00,415 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:00,416 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45350 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:00,416 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/C is initiating minor compaction (all files) 2023-04-03 13:58:00,416 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/C in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:00,416 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fa0f7134f8494d948186c9e3748afc5a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/6d5fac33ac494e26bcdc74ed53b967d2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/e5782ab1a0d4407da6b958b34738462b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5b68078a5c434d5f83b748be28480e3d] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=44.3 K 2023-04-03 13:58:00,416 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fa0f7134f8494d948186c9e3748afc5a, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680530276244 2023-04-03 13:58:00,416 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 6d5fac33ac494e26bcdc74ed53b967d2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680530276308 2023-04-03 13:58:00,417 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e5782ab1a0d4407da6b958b34738462b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1680530277424 2023-04-03 13:58:00,417 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5b68078a5c434d5f83b748be28480e3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680530278624 2023-04-03 13:58:00,422 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#C#compaction#309 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:00,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530280434, max timestamp=1680530280444Num uniques 1036; 2023-04-03 13:58:00,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-03 13:58:00,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530280434, max timestamp=1680530280444Num uniques 1036; 2023-04-03 13:58:00,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-03 13:58:00,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530280434, max timestamp=1680530280444Num uniques 1036; 2023-04-03 13:58:00,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-03 13:58:00,463 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/475bba786b4e48a4905eb399a856c0f1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/475bba786b4e48a4905eb399a856c0f1 2023-04-03 13:58:00,469 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/A of 899cf3d098e77c9ee3a377e83dee985e into 475bba786b4e48a4905eb399a856c0f1(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:00,469 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:00,469 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/A, priority=12, startTime=1680530279894; duration=0sec 2023-04-03 13:58:00,469 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:00,481 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040318a4be909a5c4b6b942060cdd6c5c5e5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040318a4be909a5c4b6b942060cdd6c5c5e5 2023-04-03 13:58:00,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680530280489, max timestamp=1680530280502Num uniques 1045; 2023-04-03 13:58:00,504 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-03 13:58:00,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680530280489, max timestamp=1680530280502Num uniques 1045; 2023-04-03 13:58:00,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-03 13:58:00,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680530280489, max timestamp=1680530280502Num uniques 1045; 2023-04-03 13:58:00,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-03 13:58:00,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530340543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530340546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530340647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530340651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530340778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530340778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530340781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,830 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/fb7869da1d36402aa2ac8f0f6179472f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fb7869da1d36402aa2ac8f0f6179472f 2023-04-03 13:58:00,833 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/C of 899cf3d098e77c9ee3a377e83dee985e into fb7869da1d36402aa2ac8f0f6179472f(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:00,834 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:00,834 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/C, priority=12, startTime=1680530279895; duration=0sec 2023-04-03 13:58:00,834 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:00,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530340852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:00,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42080 deadline: 1680530340857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:00,887 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/68acc82de7e84c0d853ac3ebe035cec2 2023-04-03 13:58:01,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:01,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530341156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:01,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530341161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,299 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/cf32d7490af9472881dcf0586557007b 2023-04-03 13:58:01,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:01,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42066 deadline: 1680530341663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530341668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,712 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/aafe5fff7c42463fbd626af73024c5d9 2023-04-03 13:58:01,717 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/68acc82de7e84c0d853ac3ebe035cec2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/68acc82de7e84c0d853ac3ebe035cec2 2023-04-03 13:58:01,721 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/68acc82de7e84c0d853ac3ebe035cec2, entries=150, sequenceid=155, filesize=25.3 K 2023-04-03 13:58:01,722 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/cf32d7490af9472881dcf0586557007b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/cf32d7490af9472881dcf0586557007b 2023-04-03 13:58:01,725 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/cf32d7490af9472881dcf0586557007b, entries=150, sequenceid=155, filesize=11.8 K 2023-04-03 13:58:01,726 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/aafe5fff7c42463fbd626af73024c5d9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/aafe5fff7c42463fbd626af73024c5d9 2023-04-03 13:58:01,729 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/aafe5fff7c42463fbd626af73024c5d9, entries=150, sequenceid=155, filesize=11.8 K 2023-04-03 13:58:01,730 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 899cf3d098e77c9ee3a377e83dee985e in 1724ms, sequenceid=155, compaction requested=false 2023-04-03 13:58:01,730 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:01,730 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:01,730 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:01,730 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:01,730 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,730 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:01,730 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:01,731 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,731 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:01,731 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:01,731 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:01,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:01,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:01,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:01,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:01,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@475c1ef8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:01,732 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:01,732 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:01,733 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:01,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:01,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:01,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:01,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:01,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:01,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:01,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:01,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:01,734 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,734 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:01,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:01,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:01,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:01,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:01,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:01,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:01,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:01,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:01,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:01,737 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:01,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:01,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:01,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:01,737 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:01,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:01,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:01,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:01,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:01,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:01,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,740 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,740 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:01,740 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:01,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:01,740 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:01,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:01,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:01,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:01,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:01,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:01,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:01,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:01,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:01,741 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:01,741 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:01,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,841 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:01,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:01,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:01,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:01,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:01,844 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:01,844 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:01,844 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:01,844 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:01,844 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,844 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:01,845 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:01,845 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:01,845 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:01,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:01,845 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:01,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:01,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:01,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:01,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:01,846 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:01,846 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:01,847 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:01,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:01,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:01,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:01,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:01,850 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=147.60 KB heapSize=416.77 KB 2023-04-03 13:58:01,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:01,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:01,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:01,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:01,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:01,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:01,865 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d3000feaa05941a488cc4c78a5979b8c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d3000feaa05941a488cc4c78a5979b8c 2023-04-03 13:58:01,872 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=264, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/5d4d2b27d8ad41298340cfbc83c46014 2023-04-03 13:58:01,881 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/407ae9a76e8e41fb8cb14d43bbfff94c 2023-04-03 13:58:02,292 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=264 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/8179f471c3504bc2a7a8f585343aab32 2023-04-03 13:58:02,297 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/5d4d2b27d8ad41298340cfbc83c46014 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/5d4d2b27d8ad41298340cfbc83c46014 2023-04-03 13:58:02,300 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/5d4d2b27d8ad41298340cfbc83c46014, entries=150, sequenceid=264, filesize=25.4 K 2023-04-03 13:58:02,300 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/407ae9a76e8e41fb8cb14d43bbfff94c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/407ae9a76e8e41fb8cb14d43bbfff94c 2023-04-03 13:58:02,303 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/407ae9a76e8e41fb8cb14d43bbfff94c, entries=150, sequenceid=264, filesize=11.9 K 2023-04-03 13:58:02,304 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/8179f471c3504bc2a7a8f585343aab32 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8179f471c3504bc2a7a8f585343aab32 2023-04-03 13:58:02,343 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8179f471c3504bc2a7a8f585343aab32, entries=150, sequenceid=264, filesize=11.9 K 2023-04-03 13:58:02,344 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~416.72 KB/426720, currentSize=0 B/0 for 899cf3d098e77c9ee3a377e83dee985e in 494ms, sequenceid=264, compaction requested=true 2023-04-03 13:58:02,344 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:02,344 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:02,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:02,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:02,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:02,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:02,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:02,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79132093[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:02,350 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:02,350 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,351 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:02,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:02,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:02,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:02,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:02,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:02,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:02,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:02,355 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:02,355 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:02,355 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:02,355 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:02,355 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:02,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:02,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,359 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,361 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:02,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:02,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:02,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:02,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:02,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:02,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:02,361 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:02,361 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,362 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:02,362 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:02,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:02,462 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:02,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:02,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:02,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:02,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:02,466 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:02,466 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:02,466 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:02,466 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:02,467 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,467 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,467 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,467 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:02,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:02,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:02,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:02,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:02,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:02,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:02,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:02,474 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:02,474 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:02,474 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:02,486 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:02,486 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:02,498 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:02,498 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:02,498 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:02,498 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:02,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:02,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:02,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:02,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:02,500 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,500 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,500 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,500 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:02,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f1fd1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:02,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:02,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,502 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,502 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:02,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,503 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:02,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:02,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:02,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:02,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:02,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:02,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:02,507 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:02,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:02,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:02,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:02,507 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:02,508 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,508 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,508 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:02,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:02,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:02,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:02,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:02,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:02,513 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:02,513 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:02,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,513 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:02,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:02,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:02,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:02,613 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:02,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:02,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:02,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:02,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:02,624 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:02,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:02,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:02,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:02,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,625 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,626 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:02,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:02,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:02,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:02,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:02,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:02,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:02,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:02,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:02,628 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:02,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:02,629 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:02,629 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:02,629 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:02,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:02,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:02,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:02,629 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:02,630 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,631 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,631 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:02,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:02,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a489b4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:02,633 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:02,633 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,635 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:02,635 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:02,635 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:02,635 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,635 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:02,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:02,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:02,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:02,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:02,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:02,643 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:02,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:02,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:02,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:02,643 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:02,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:02,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:02,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,646 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,653 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,653 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:02,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,654 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:02,653 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:02,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:02,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:02,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:02,654 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:02,654 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:02,654 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:02,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:02,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:02,754 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:02,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:02,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:02,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:02,786 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:02,786 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:02,786 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:02,786 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:02,787 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,787 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,787 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,787 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:02,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:02,788 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:02,788 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:02,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:02,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:02,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:02,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:02,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:02,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:02,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:02,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:02,802 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:02,803 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:02,803 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:02,803 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-03 13:58:02,803 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:02,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:02,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:02,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:02,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:02,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:02,844 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:02,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:02,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530282929, max timestamp=1680530282961Num uniques 859; 2023-04-03 13:58:02,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530282929, max timestamp=1680530282961Num uniques 859; 2023-04-03 13:58:02,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:02,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:02,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530282929, max timestamp=1680530282961Num uniques 859; 2023-04-03 13:58:02,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:03,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9968, totalHeapSize=22912, min timestamp=1680530282990, max timestamp=1680530283010Num uniques 1068; 2023-04-03 13:58:03,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9968, totalHeapSize=22912, min timestamp=1680530282990, max timestamp=1680530283010Num uniques 1068; 2023-04-03 13:58:03,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48898, new segment data size=9968, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 13:58:03,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48898, new segment data size=9968, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 13:58:03,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530282990, max timestamp=1680530283010Num uniques 1068; 2023-04-03 13:58:03,014 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 13:58:03,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530283043, max timestamp=1680530283074Num uniques 1077; 2023-04-03 13:58:03,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530283043, max timestamp=1680530283074Num uniques 1077; 2023-04-03 13:58:03,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:03,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:03,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530283043, max timestamp=1680530283076Num uniques 1077; 2023-04-03 13:58:03,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:03,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680530283094, max timestamp=1680530283128Num uniques 1036; 2023-04-03 13:58:03,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-03 13:58:03,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680530283094, max timestamp=1680530283128Num uniques 1036; 2023-04-03 13:58:03,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-03 13:58:03,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1680530283094, max timestamp=1680530283128Num uniques 1036; 2023-04-03 13:58:03,135 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-03 13:58:03,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680530283140, max timestamp=1680530283201Num uniques 1145; 2023-04-03 13:58:03,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680530283140, max timestamp=1680530283201Num uniques 1145; 2023-04-03 13:58:03,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52430, new segment data size=8920, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-03 13:58:03,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8920, totalHeapSize=20520, min timestamp=1680530283140, max timestamp=1680530283201Num uniques 1145; 2023-04-03 13:58:03,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52430, new segment data size=8920, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-03 13:58:03,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=52430, new segment data size=8920, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-03 13:58:03,295 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040357b7f16d862b4585be7833e284f7e8ce to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040357b7f16d862b4585be7833e284f7e8ce 2023-04-03 13:58:03,310 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/3ee323f0b76943e89e60b4594a8c5225 2023-04-03 13:58:03,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1680530283293, max timestamp=1680530283316Num uniques 1054; 2023-04-03 13:58:03,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1680530283293, max timestamp=1680530283316Num uniques 1054; 2023-04-03 13:58:03,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=16210, suffix heap size=137584,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=354 2023-04-03 13:58:03,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=16210, suffix heap size=137584,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=354 2023-04-03 13:58:03,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1680530283293, max timestamp=1680530283316Num uniques 1054; 2023-04-03 13:58:03,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=16210, suffix heap size=137584,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=354 2023-04-03 13:58:03,328 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/6dabae4e2293453881bfe7388b715f3f 2023-04-03 13:58:03,346 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/bea9cf9ccf2b40e2a91b79a37c97b609 2023-04-03 13:58:03,349 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/3ee323f0b76943e89e60b4594a8c5225 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3ee323f0b76943e89e60b4594a8c5225 2023-04-03 13:58:03,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680530283337, max timestamp=1680530283350Num uniques 1213; 2023-04-03 13:58:03,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680530283337, max timestamp=1680530283350Num uniques 1213; 2023-04-03 13:58:03,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680530283337, max timestamp=1680530283350Num uniques 1213; 2023-04-03 13:58:03,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=55548, new segment data size=12038, suffix heap size=154120,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1213, new segment cells count=263 2023-04-03 13:58:03,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=55548, new segment data size=12038, suffix heap size=154120,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1213, new segment cells count=263 2023-04-03 13:58:03,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=55548, new segment data size=12038, suffix heap size=154120,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1213, new segment cells count=263 2023-04-03 13:58:03,355 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3ee323f0b76943e89e60b4594a8c5225, entries=100, sequenceid=271, filesize=18.6 K 2023-04-03 13:58:03,362 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/6dabae4e2293453881bfe7388b715f3f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/6dabae4e2293453881bfe7388b715f3f 2023-04-03 13:58:03,367 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/6dabae4e2293453881bfe7388b715f3f, entries=100, sequenceid=271, filesize=9.5 K 2023-04-03 13:58:03,367 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/bea9cf9ccf2b40e2a91b79a37c97b609 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/bea9cf9ccf2b40e2a91b79a37c97b609 2023-04-03 13:58:03,374 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/bea9cf9ccf2b40e2a91b79a37c97b609, entries=100, sequenceid=271, filesize=9.5 K 2023-04-03 13:58:03,375 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=93.93 KB/96180 for 899cf3d098e77c9ee3a377e83dee985e in 572ms, sequenceid=271, compaction requested=true 2023-04-03 13:58:03,375 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:03,375 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,375 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:03,375 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:03,375 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,375 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:03,375 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:03,376 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,376 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:03,379 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:03,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a8fc26[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:03,379 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,380 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,380 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:03,380 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:03,380 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:03,380 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,381 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:03,381 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:03,381 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:03,381 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:03,381 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:03,382 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=100.63 KB heapSize=275.63 KB 2023-04-03 13:58:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:03,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:03,387 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:03,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:03,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:03,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:03,387 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:03,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:03,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,396 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:03,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:03,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:03,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:03,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:03,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:03,402 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,402 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:03,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,402 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:03,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:03,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530343408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530343408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530343408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530343409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530343410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,502 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:03,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:03,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:03,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:03,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:03,510 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:03,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:03,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:03,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:03,511 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,511 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:03,512 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:03,514 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,514 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:03,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:03,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:03,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:03,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:03,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:03,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530343522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530343522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530343523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,534 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:03,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530343524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,534 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:03,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,534 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530343524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,534 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:03,534 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:03,535 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,535 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:03,535 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:03,535 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,535 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:03,535 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:03,536 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,536 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,537 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,537 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:03,538 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:03,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42df4758[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:03,538 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,539 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,539 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,539 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:03,539 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:03,539 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:03,539 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:03,540 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,540 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,540 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:03,540 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:03,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:03,543 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:03,543 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:03,543 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:03,543 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:03,543 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:03,544 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,544 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,544 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:03,545 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,546 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,551 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:03,551 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:03,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:03,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:03,551 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,551 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,551 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,551 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:03,551 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:03,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:03,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:03,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:03,552 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,552 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,552 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,552 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:03,552 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:03,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:03,652 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:03,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:03,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:03,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:03,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:03,678 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:03,678 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:03,678 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:03,678 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:03,679 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,679 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,680 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,681 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:03,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:03,681 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,682 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:03,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:03,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:03,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:03,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:03,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:03,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:03,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:03,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:03,694 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,694 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:03,694 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:03,694 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:03,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:03,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:03,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:03,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,696 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:03,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b876945[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:03,698 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:03,698 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,699 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,699 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,699 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:03,699 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:03,699 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:03,699 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:03,701 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:03,701 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:03,702 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:03,702 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:03,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:03,718 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:03,718 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:03,718 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:03,718 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:03,718 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:03,719 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,719 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:03,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,720 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,721 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,722 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,723 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,724 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:03,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:03,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:03,724 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:03,724 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:03,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:03,724 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:03,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:03,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:03,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:03,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:03,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530343726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530343727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530343733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530343735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:03,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530343735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,801 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d154bdcb78cc45b895b12e2cd1716543 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d154bdcb78cc45b895b12e2cd1716543 2023-04-03 13:58:03,824 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:03,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:03,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:03,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:03,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:03,830 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:03,830 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:03,830 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:03,830 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:03,830 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=404, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/316bc924d46342568cb3ae2ea3536c1e 2023-04-03 13:58:03,831 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,831 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,831 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:03,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:03,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:03,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:03,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:03,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:03,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:03,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:03,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:03,838 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,838 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:03,838 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:03,838 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:03,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:03,844 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:03,846 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,846 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:03,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a4faea8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:03,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:03,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,854 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/738ba42aa5fd43c7a299594da668bb6c 2023-04-03 13:58:03,860 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:03,861 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:03,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:03,868 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:03,868 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:03,868 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,868 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:03,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:03,881 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:03,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:03,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:03,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:03,881 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:03,882 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,882 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,882 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:03,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:03,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:03,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:03,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:03,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:03,894 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:03,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:03,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:03,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:03,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:03,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:03,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,894 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:03,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:03,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:03,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:03,894 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:03,894 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:03,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/5df93dafdf764c708044230c1324be40 2023-04-03 13:58:03,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/316bc924d46342568cb3ae2ea3536c1e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/316bc924d46342568cb3ae2ea3536c1e 2023-04-03 13:58:03,933 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/316bc924d46342568cb3ae2ea3536c1e, entries=300, sequenceid=404, filesize=46.1 K 2023-04-03 13:58:03,934 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/738ba42aa5fd43c7a299594da668bb6c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/738ba42aa5fd43c7a299594da668bb6c 2023-04-03 13:58:03,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/738ba42aa5fd43c7a299594da668bb6c, entries=150, sequenceid=404, filesize=11.9 K 2023-04-03 13:58:03,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/5df93dafdf764c708044230c1324be40 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5df93dafdf764c708044230c1324be40 2023-04-03 13:58:03,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5df93dafdf764c708044230c1324be40, entries=150, sequenceid=404, filesize=11.9 K 2023-04-03 13:58:03,968 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~275.58 KB/282192, currentSize=80.51 KB/82440 for 899cf3d098e77c9ee3a377e83dee985e in 586ms, sequenceid=404, compaction requested=true 2023-04-03 13:58:03,968 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:03,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:03,968 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 13:58:03,968 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 13:58:03,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:03,968 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:03,969 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 144144 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 13:58:03,969 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58324 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 13:58:03,969 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/A is initiating minor compaction (all files) 2023-04-03 13:58:03,969 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/B is initiating minor compaction (all files) 2023-04-03 13:58:03,970 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/A in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,970 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/B in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:03,970 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/ea371f743761460fbaccdecc690265d7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/cf32d7490af9472881dcf0586557007b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/407ae9a76e8e41fb8cb14d43bbfff94c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/6dabae4e2293453881bfe7388b715f3f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/738ba42aa5fd43c7a299594da668bb6c] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=57.0 K 2023-04-03 13:58:03,970 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/475bba786b4e48a4905eb399a856c0f1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/68acc82de7e84c0d853ac3ebe035cec2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/5d4d2b27d8ad41298340cfbc83c46014, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3ee323f0b76943e89e60b4594a8c5225, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/316bc924d46342568cb3ae2ea3536c1e] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=140.8 K 2023-04-03 13:58:03,970 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ea371f743761460fbaccdecc690265d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680530278624 2023-04-03 13:58:03,970 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 475bba786b4e48a4905eb399a856c0f1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680530278624 2023-04-03 13:58:03,970 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting cf32d7490af9472881dcf0586557007b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680530278910 2023-04-03 13:58:03,971 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 68acc82de7e84c0d853ac3ebe035cec2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680530278910 2023-04-03 13:58:03,972 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 407ae9a76e8e41fb8cb14d43bbfff94c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680530280532 2023-04-03 13:58:03,972 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5d4d2b27d8ad41298340cfbc83c46014, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680530280532 2023-04-03 13:58:03,972 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 6dabae4e2293453881bfe7388b715f3f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680530282701 2023-04-03 13:58:03,972 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3ee323f0b76943e89e60b4594a8c5225, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680530282701 2023-04-03 13:58:03,973 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 316bc924d46342568cb3ae2ea3536c1e, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680530283362 2023-04-03 13:58:03,973 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 738ba42aa5fd43c7a299594da668bb6c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680530283365 2023-04-03 13:58:03,990 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#B#compaction#321 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:03,994 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:03,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:03,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:04,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:04,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:04,012 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:04,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:04,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:04,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:04,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:04,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:04,013 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:04,013 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:04,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:04,014 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,014 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:04,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:04,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:04,020 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#A#compaction#322 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:04,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:04,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:04,023 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:04,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:04,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:04,039 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:04,039 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:04,039 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 13:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:04,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:04,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:04,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:04,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530344087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530344095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530344096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530344101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530344102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530344204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530344208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530344208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530344213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530344216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530344415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,421 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/e07642945df8421dba2d4ba619e343bc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e07642945df8421dba2d4ba619e343bc 2023-04-03 13:58:04,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530344419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530344419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530344419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,427 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/B of 899cf3d098e77c9ee3a377e83dee985e into e07642945df8421dba2d4ba619e343bc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:04,427 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:04,427 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/B, priority=11, startTime=1680530283968; duration=0sec 2023-04-03 13:58:04,427 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:04,427 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 13:58:04,428 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58324 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 13:58:04,428 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/C is initiating minor compaction (all files) 2023-04-03 13:58:04,428 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/C in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:04,428 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fb7869da1d36402aa2ac8f0f6179472f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/aafe5fff7c42463fbd626af73024c5d9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8179f471c3504bc2a7a8f585343aab32, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/bea9cf9ccf2b40e2a91b79a37c97b609, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5df93dafdf764c708044230c1324be40] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=57.0 K 2023-04-03 13:58:04,428 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fb7869da1d36402aa2ac8f0f6179472f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680530278624 2023-04-03 13:58:04,429 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting aafe5fff7c42463fbd626af73024c5d9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680530278910 2023-04-03 13:58:04,429 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 8179f471c3504bc2a7a8f585343aab32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=264, earliestPutTs=1680530280532 2023-04-03 13:58:04,429 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bea9cf9ccf2b40e2a91b79a37c97b609, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680530282701 2023-04-03 13:58:04,429 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5df93dafdf764c708044230c1324be40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680530283365 2023-04-03 13:58:04,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530344427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,452 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#C#compaction#324 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:04,464 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/0f25017f78484f7abec47e8d5cb0e76c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0f25017f78484f7abec47e8d5cb0e76c 2023-04-03 13:58:04,471 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/C of 899cf3d098e77c9ee3a377e83dee985e into 0f25017f78484f7abec47e8d5cb0e76c(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:04,471 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:04,471 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/C, priority=11, startTime=1680530283968; duration=0sec 2023-04-03 13:58:04,471 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:04,532 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036c9734587ff54740b92cd2f31603c6b1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036c9734587ff54740b92cd2f31603c6b1 2023-04-03 13:58:04,533 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/99a6210209464c2ab3d71d2c9b3c8e07 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/99a6210209464c2ab3d71d2c9b3c8e07 2023-04-03 13:58:04,536 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/A of 899cf3d098e77c9ee3a377e83dee985e into 99a6210209464c2ab3d71d2c9b3c8e07(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:04,537 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:04,537 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/A, priority=11, startTime=1680530283968; duration=0sec 2023-04-03 13:58:04,537 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:04,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530344718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530344726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530344727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530344728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:04,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530344736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:04,937 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=419, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/cb3cb021f2334799ac72ed2e2ff3a968 2023-04-03 13:58:05,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:05,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530345223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:05,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530345232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530345233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:05,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530345234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,243 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:05,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530345241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,344 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/d9e15eb01fcd4a71b895de4cf0de2b7f 2023-04-03 13:58:05,752 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/4f0131092cd84020aa4321ab43eaa896 2023-04-03 13:58:05,755 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/cb3cb021f2334799ac72ed2e2ff3a968 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb3cb021f2334799ac72ed2e2ff3a968 2023-04-03 13:58:05,759 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb3cb021f2334799ac72ed2e2ff3a968, entries=150, sequenceid=419, filesize=25.4 K 2023-04-03 13:58:05,759 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/d9e15eb01fcd4a71b895de4cf0de2b7f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d9e15eb01fcd4a71b895de4cf0de2b7f 2023-04-03 13:58:05,762 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d9e15eb01fcd4a71b895de4cf0de2b7f, entries=150, sequenceid=419, filesize=11.9 K 2023-04-03 13:58:05,763 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/4f0131092cd84020aa4321ab43eaa896 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4f0131092cd84020aa4321ab43eaa896 2023-04-03 13:58:05,765 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4f0131092cd84020aa4321ab43eaa896, entries=150, sequenceid=419, filesize=11.9 K 2023-04-03 13:58:05,766 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 899cf3d098e77c9ee3a377e83dee985e in 1727ms, sequenceid=419, compaction requested=false 2023-04-03 13:58:05,766 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:05,766 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:05,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:05,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:05,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:05,766 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:05,767 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,767 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:05,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:05,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:05,768 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:05,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:05,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:05,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fe3280b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:05,770 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:05,770 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:05,771 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:05,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:05,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:05,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:05,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:05,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:05,771 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,771 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:05,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:05,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:05,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:05,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:05,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:05,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:05,774 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:05,774 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:05,774 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:05,774 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:05,774 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:05,775 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,775 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,775 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:05,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:05,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:05,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:05,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:05,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:05,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:05,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:05,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:05,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:05,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:05,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:05,779 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:05,779 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:05,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:05,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:05,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:05,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:05,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:05,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,880 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:05,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:05,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:05,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:05,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:05,882 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:05,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:05,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:05,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:05,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:05,883 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:05,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:05,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:05,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:05,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:05,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:05,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:05,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:05,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:05,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:05,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:05,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:05,885 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:05,885 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:05,885 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 13:58:05,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:05,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:05,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:05,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:05,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:05,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:06,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:06,235 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:06,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530346252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530346254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530346256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530346256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530346258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,298 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403e7d656d210aa43a0b96d55d8488b604e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e7d656d210aa43a0b96d55d8488b604e 2023-04-03 13:58:06,303 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=439, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/2b483dde16634848bd895987a476db5a 2023-04-03 13:58:06,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530346362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530346362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530346364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530346368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530346568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530346568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530346569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530346574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,710 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/9ce476cce225447c981c39aa4b850239 2023-04-03 13:58:06,719 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/8df37611b60c4f5294db2f5516704ee8 2023-04-03 13:58:06,723 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/2b483dde16634848bd895987a476db5a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/2b483dde16634848bd895987a476db5a 2023-04-03 13:58:06,726 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/2b483dde16634848bd895987a476db5a, entries=150, sequenceid=439, filesize=25.4 K 2023-04-03 13:58:06,726 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/9ce476cce225447c981c39aa4b850239 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/9ce476cce225447c981c39aa4b850239 2023-04-03 13:58:06,729 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/9ce476cce225447c981c39aa4b850239, entries=150, sequenceid=439, filesize=11.9 K 2023-04-03 13:58:06,729 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/8df37611b60c4f5294db2f5516704ee8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8df37611b60c4f5294db2f5516704ee8 2023-04-03 13:58:06,732 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8df37611b60c4f5294db2f5516704ee8, entries=150, sequenceid=439, filesize=11.9 K 2023-04-03 13:58:06,732 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 899cf3d098e77c9ee3a377e83dee985e in 847ms, sequenceid=439, compaction requested=true 2023-04-03 13:58:06,732 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:06,732 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:06,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:06,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:06,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:06,732 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:06,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:06,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:06,733 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:06,734 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:06,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:06,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:06,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473e5528[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:06,736 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:06,736 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:06,736 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:06,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:06,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:06,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:06,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:06,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:06,736 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,736 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:06,737 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:06,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:06,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:06,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:06,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:06,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:06,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:06,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:06,739 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:06,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:06,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:06,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:06,739 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:06,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:06,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:06,740 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:06,741 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:06,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:06,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:06,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:06,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:06,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:06,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:06,743 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:06,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:06,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:06,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:06,743 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:06,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,844 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:06,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:06,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:06,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:06,846 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:06,846 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:06,846 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:06,846 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:06,847 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,847 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:06,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,848 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:06,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:06,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:06,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:06,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:06,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:06,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:06,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:06,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:06,849 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:06,849 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:06,849 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:06,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:06,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:06,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:06,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:06,850 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 13:58:06,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:06,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:06,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:06,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:06,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:06,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:06,862 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304034c94a403542c498694b17ea9444b80c0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034c94a403542c498694b17ea9444b80c0 2023-04-03 13:58:06,866 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/3d6f6d47d53a4e4d82a303496dca57b1 2023-04-03 13:58:06,878 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/65ec0324266c42fa85c86ccb98726b83 2023-04-03 13:58:06,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:06,879 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:06,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530346893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530346893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530346897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:06,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:06,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530346898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530346999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530347001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,005 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530347002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530347005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530347206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530347207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530347207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530347210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,286 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/3ab82b17ee6a43a5b46f0967fe772739 2023-04-03 13:58:07,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/3d6f6d47d53a4e4d82a303496dca57b1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3d6f6d47d53a4e4d82a303496dca57b1 2023-04-03 13:58:07,293 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3d6f6d47d53a4e4d82a303496dca57b1, entries=150, sequenceid=454, filesize=25.4 K 2023-04-03 13:58:07,293 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/65ec0324266c42fa85c86ccb98726b83 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/65ec0324266c42fa85c86ccb98726b83 2023-04-03 13:58:07,299 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/65ec0324266c42fa85c86ccb98726b83, entries=150, sequenceid=454, filesize=11.9 K 2023-04-03 13:58:07,300 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/3ab82b17ee6a43a5b46f0967fe772739 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/3ab82b17ee6a43a5b46f0967fe772739 2023-04-03 13:58:07,304 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/3ab82b17ee6a43a5b46f0967fe772739, entries=150, sequenceid=454, filesize=11.9 K 2023-04-03 13:58:07,305 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 899cf3d098e77c9ee3a377e83dee985e in 455ms, sequenceid=454, compaction requested=true 2023-04-03 13:58:07,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:07,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:07,305 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:07,305 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:07,305 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,305 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:07,305 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:07,306 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,306 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:07,306 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:07,306 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:07,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:07,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47f3c1cb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:07,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:07,307 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:07,308 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:07,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:07,308 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:07,308 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:07,308 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:07,308 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,308 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:07,309 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:07,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:07,309 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:07,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:07,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:07,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:07,319 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:07,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:07,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:07,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:07,320 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:07,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:07,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:07,327 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:07,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:07,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:07,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:07,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:07,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:07,331 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:07,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:07,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:07,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:07,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:07,331 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:07,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:07,331 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:07,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:07,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:07,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:07,431 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:07,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:07,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:07,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:07,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:07,434 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:07,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:07,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:07,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:07,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:07,435 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,435 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:07,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:07,435 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,435 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:07,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:07,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:07,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:07,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:07,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:07,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:07,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:07,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:07,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:07,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:07,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:07,436 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:07,436 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:07,436 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 13:58:07,436 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:07,436 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:07,437 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:07,437 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:07,437 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:07,437 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:07,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:07,515 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:07,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530347530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530347530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530347531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,537 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530347532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530347637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530347637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530347637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530347637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530347842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530347842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530347842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:07,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530347842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:07,850 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403fb379fc7de5843d7b63d46301e52cb58 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fb379fc7de5843d7b63d46301e52cb58 2023-04-03 13:58:08,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530348148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530348148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530348149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530348149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,254 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=471, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/9bf5cb7a4c0e4ba980f1e3adee8563f5 2023-04-03 13:58:08,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530348260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,266 DEBUG [Thread-2309] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:58:08,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530348654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42110 deadline: 1680530348656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42084 deadline: 1680530348656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42096 deadline: 1680530348657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:08,662 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/77fa0f432e434679b14c300ff40906bc 2023-04-03 13:58:09,072 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=471 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/379c12dd6ef54aa1aaf1ea54618a781c 2023-04-03 13:58:09,076 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/9bf5cb7a4c0e4ba980f1e3adee8563f5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9bf5cb7a4c0e4ba980f1e3adee8563f5 2023-04-03 13:58:09,079 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9bf5cb7a4c0e4ba980f1e3adee8563f5, entries=150, sequenceid=471, filesize=25.4 K 2023-04-03 13:58:09,079 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-03 13:58:09,080 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/77fa0f432e434679b14c300ff40906bc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/77fa0f432e434679b14c300ff40906bc 2023-04-03 13:58:09,082 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/77fa0f432e434679b14c300ff40906bc, entries=150, sequenceid=471, filesize=11.9 K 2023-04-03 13:58:09,083 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/379c12dd6ef54aa1aaf1ea54618a781c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/379c12dd6ef54aa1aaf1ea54618a781c 2023-04-03 13:58:09,086 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/379c12dd6ef54aa1aaf1ea54618a781c, entries=150, sequenceid=471, filesize=11.9 K 2023-04-03 13:58:09,086 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 899cf3d098e77c9ee3a377e83dee985e in 1650ms, sequenceid=471, compaction requested=true 2023-04-03 13:58:09,086 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:09,086 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:09,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:09,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:09,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:09,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:09,087 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:09,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:09,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:09,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:09,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:09,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20ae0d73[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:09,090 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:09,090 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:09,090 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:09,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:09,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:09,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:09,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:09,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:09,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:09,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:09,091 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:09,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:09,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:09,093 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:09,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:09,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:09,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:09,093 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:09,094 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,094 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:09,094 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:09,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:09,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:09,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:09,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:09,097 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:09,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:09,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:09,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:09,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,097 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:09,097 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:09,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:09,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:09,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,197 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:09,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:09,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:09,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:09,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:09,201 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:09,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:09,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:09,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:09,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:09,202 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,202 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:09,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:09,202 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,202 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:09,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:09,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:09,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:09,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:09,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:09,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:09,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:09,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:09,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:09,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:09,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:09,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:09,203 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 13:58:09,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:09,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:09,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:09,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:09,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:09,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:09,621 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040307e780594abf4105875eb852492a8a20 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040307e780594abf4105875eb852492a8a20 2023-04-03 13:58:09,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:09,665 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:09,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:09,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530349693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:09,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530349694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:09,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530349694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:09,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530349701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:09,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530349802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:09,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530349802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:09,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530349802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:09,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:09,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530349802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530350007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530350007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530350007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530350008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,046 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=486, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/cb5d0b7a74d44593bf978ff465383246 2023-04-03 13:58:10,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530350313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530350313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530350314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530350314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,454 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/86cee91554c14c50b0a27b280fec74ad 2023-04-03 13:58:10,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530350818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530350819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530350819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:10,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530350825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,867 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/366b6dee003949f78a46fd5379c5346c 2023-04-03 13:58:10,870 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/cb5d0b7a74d44593bf978ff465383246 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb5d0b7a74d44593bf978ff465383246 2023-04-03 13:58:10,873 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb5d0b7a74d44593bf978ff465383246, entries=150, sequenceid=486, filesize=25.4 K 2023-04-03 13:58:10,875 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/86cee91554c14c50b0a27b280fec74ad as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/86cee91554c14c50b0a27b280fec74ad 2023-04-03 13:58:10,879 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/86cee91554c14c50b0a27b280fec74ad, entries=150, sequenceid=486, filesize=11.9 K 2023-04-03 13:58:10,879 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/366b6dee003949f78a46fd5379c5346c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/366b6dee003949f78a46fd5379c5346c 2023-04-03 13:58:10,882 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/366b6dee003949f78a46fd5379c5346c, entries=150, sequenceid=486, filesize=11.9 K 2023-04-03 13:58:10,882 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 899cf3d098e77c9ee3a377e83dee985e in 1679ms, sequenceid=486, compaction requested=true 2023-04-03 13:58:10,882 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:10,882 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:10,882 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:10,882 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:10,882 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,882 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:10,882 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:10,883 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,883 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:10,883 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:10,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:10,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:10,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ebca1d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:10,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:10,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:10,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:10,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:10,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:10,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:10,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:10,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:10,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:10,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:10,886 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:10,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:10,888 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:10,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:10,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:10,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:10,888 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:10,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:10,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:10,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:10,889 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:10,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:10,889 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:10,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:10,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:10,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:10,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:10,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:10,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:10,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:10,892 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:10,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:10,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:10,892 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:10,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:10,893 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:10,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:10,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:10,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:10,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:10,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:10,993 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:10,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:11,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:11,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:11,002 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:11,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:11,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:11,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:11,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,002 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:11,003 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:11,003 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:11,003 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,003 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:11,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:11,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:11,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:11,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:11,004 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:11,004 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:11,004 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:11,004 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:11,004 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:11,004 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:11,004 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:11,005 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 13:58:11,005 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:11,005 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:11,005 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:11,005 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:11,005 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:11,005 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:11,026 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403118f88ab37ae46fc8ba3879c11dab171 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403118f88ab37ae46fc8ba3879c11dab171 2023-04-03 13:58:11,439 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=503, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/261db5a298c744e1a9f98687e4fa69ce 2023-04-03 13:58:11,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:11,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:11,848 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/c2d71bfd92f84e21bd381b9b93f39b43 2023-04-03 13:58:11,857 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/9a55283720134ddfad6d8ae4e6914cb5 2023-04-03 13:58:11,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:11,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530351851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:11,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530351851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:11,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530351851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,864 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/261db5a298c744e1a9f98687e4fa69ce as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/261db5a298c744e1a9f98687e4fa69ce 2023-04-03 13:58:11,870 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/261db5a298c744e1a9f98687e4fa69ce, entries=150, sequenceid=503, filesize=25.4 K 2023-04-03 13:58:11,871 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/c2d71bfd92f84e21bd381b9b93f39b43 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/c2d71bfd92f84e21bd381b9b93f39b43 2023-04-03 13:58:11,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:11,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530351863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,884 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/c2d71bfd92f84e21bd381b9b93f39b43, entries=150, sequenceid=503, filesize=11.9 K 2023-04-03 13:58:11,885 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/9a55283720134ddfad6d8ae4e6914cb5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9a55283720134ddfad6d8ae4e6914cb5 2023-04-03 13:58:11,888 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9a55283720134ddfad6d8ae4e6914cb5, entries=150, sequenceid=503, filesize=11.9 K 2023-04-03 13:58:11,889 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 899cf3d098e77c9ee3a377e83dee985e in 885ms, sequenceid=503, compaction requested=true 2023-04-03 13:58:11,889 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:11,889 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:11,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:11,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:11,890 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,890 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:11,890 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:11,890 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:11,891 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:11,891 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:11,891 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:11,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:11,892 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:11,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@734940fd[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:11,892 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:11,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:11,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:11,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:11,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:11,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:11,893 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,893 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:11,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:11,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:11,894 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:11,894 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:11,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:11,895 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:11,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:11,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:11,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:11,895 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:11,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:11,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:11,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:11,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:11,898 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,899 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:11,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:11,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:11,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:11,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:11,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:11,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:11,901 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:11,901 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:11,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:11,902 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:11,902 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:11,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:11,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 13:58:11,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:11,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:11,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:11,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:11,974 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:11,974 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:12,002 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,004 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,004 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,004 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,006 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,007 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,007 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,008 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,008 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,008 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,009 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aed2ac6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,009 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,010 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,010 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,012 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,012 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,013 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530352000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,013 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,013 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,013 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,014 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,015 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530352001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530352001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,016 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,016 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,016 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,016 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530352013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530352114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,116 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,119 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,120 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:12,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,121 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,121 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,121 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,121 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,121 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,122 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,123 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7af87f7d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,123 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,124 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,124 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,124 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,125 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,125 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,125 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,125 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530352116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530352117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530352117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,129 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,129 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,129 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,129 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,129 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,138 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,138 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,138 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,138 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,143 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,143 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,143 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,143 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,143 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,143 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,143 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,143 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,143 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,144 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,244 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,246 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,246 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,246 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,247 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,247 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,247 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,248 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,248 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,248 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,248 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:12,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,249 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,250 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,250 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,250 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,250 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,250 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,250 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,251 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,251 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,252 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625bf91d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,252 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,252 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,252 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,253 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,253 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,253 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,253 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,255 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,255 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,255 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,255 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,255 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,256 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,256 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,256 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,257 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,259 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,260 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,260 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,260 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,260 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,260 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,260 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,261 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,261 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,261 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,261 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,261 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,261 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42080 deadline: 1680530352305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,310 DEBUG [Thread-2309] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8207 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:58:12,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530352317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530352326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530352327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530352327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,361 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,368 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,369 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,369 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,369 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,370 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,370 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,370 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:12,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,374 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,374 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,374 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,377 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,377 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,377 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,377 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,377 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,379 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,379 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,380 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,380 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,383 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b2c0303[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,383 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,384 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403c94e14d2267147b78384c89dbf976015 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c94e14d2267147b78384c89dbf976015 2023-04-03 13:58:12,388 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,388 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,388 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,389 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,393 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,393 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,393 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,394 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,394 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,394 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,395 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,397 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,398 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,404 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=519, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/194a52e73bbb4af7b592d2e375ab3669 2023-04-03 13:58:12,498 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,501 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,502 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:12,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,504 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d4b9ba[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,506 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,506 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,506 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,506 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,506 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,506 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,506 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,506 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,507 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,507 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,507 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,507 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,509 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,509 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,509 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,510 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,511 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,511 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,511 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,512 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,513 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,514 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,514 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,514 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,514 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,514 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,614 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,617 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,617 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,617 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,617 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,618 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:12,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,619 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,619 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,620 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,620 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,620 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,620 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,620 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,620 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,620 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21ee3e4a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,623 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,624 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,625 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,626 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,626 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,627 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,627 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,628 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530352620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,629 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,629 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,629 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530352635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530352637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:12,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530352637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,730 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,733 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,734 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:12,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,736 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c40a87b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,738 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,742 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,742 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,744 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,746 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,746 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,746 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/b14404e6291c4513853daeb46f69027d 2023-04-03 13:58:12,846 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,848 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,850 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,850 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,850 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,850 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:12,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,851 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,851 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,851 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,851 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,851 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,851 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,852 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,852 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,852 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,852 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,852 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,852 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,852 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,852 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,853 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,853 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,853 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,853 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27cf94eb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,854 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,855 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,858 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,858 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,858 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,859 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,859 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,860 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,860 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,860 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,862 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,863 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,863 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,964 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:12,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:12,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:12,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:12,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:12,966 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:12,966 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,967 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:12,967 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:12,967 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,967 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,968 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,968 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:12,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:12,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:12,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:12,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:12,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:12,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:12,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:12,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:12,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:12,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:12,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:12,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:12,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:12,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:12,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:12,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:12,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:12,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aebe0c1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:12,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:12,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,974 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:12,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:12,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:12,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:12,975 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:12,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:12,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:12,977 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:12,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:12,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:12,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:12,977 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:12,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:12,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:12,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:12,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:12,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:12,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:12,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:12,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:12,980 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:12,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:12,980 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:12,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:12,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:13,081 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:13,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:13,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:13,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:13,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:13,083 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:13,083 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:13,083 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:13,083 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:13,084 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,084 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,084 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,084 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:13,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:13,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:13,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:13,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:13,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:13,086 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:13,086 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:13,086 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:13,086 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:13,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:13,087 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:13,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:13,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:13,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:13,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a2791b8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:13,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,089 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,089 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:13,089 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:13,089 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:13,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:13,090 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:13,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:13,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:13,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:13,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:13,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:13,092 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:13,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:13,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:13,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:13,092 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:13,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:13,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:13,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,096 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:13,096 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:13,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:13,097 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:13,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:13,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:13,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:13,097 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:13,097 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:13,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:13,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:13,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530353130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:13,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530353141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530353143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:13,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530353143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,197 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:13,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:13,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:13,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:13,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:13,200 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:13,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:13,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:13,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:13,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,201 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:13,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:13,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:13,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:13,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:13,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:13,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:13,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:13,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:13,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:13,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:13,204 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:13,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:13,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:13,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:13,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53bdfbf7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:13,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:13,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,206 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:13,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:13,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:13,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:13,206 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:13,207 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:13,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:13,207 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:13,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:13,208 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:13,208 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:13,208 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:13,208 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:13,208 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:13,209 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:13,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:13,209 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:13,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:13,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:13,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:13,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:13,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:13,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:13,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:13,212 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:13,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:13,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:13,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:13,212 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:13,213 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:13,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:13,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/0aa728f7de844b3ea5de4902d2988e2b 2023-04-03 13:58:13,224 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/194a52e73bbb4af7b592d2e375ab3669 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/194a52e73bbb4af7b592d2e375ab3669 2023-04-03 13:58:13,227 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/194a52e73bbb4af7b592d2e375ab3669, entries=200, sequenceid=519, filesize=32.3 K 2023-04-03 13:58:13,227 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/b14404e6291c4513853daeb46f69027d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/b14404e6291c4513853daeb46f69027d 2023-04-03 13:58:13,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/b14404e6291c4513853daeb46f69027d, entries=150, sequenceid=519, filesize=11.9 K 2023-04-03 13:58:13,231 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/0aa728f7de844b3ea5de4902d2988e2b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0aa728f7de844b3ea5de4902d2988e2b 2023-04-03 13:58:13,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0aa728f7de844b3ea5de4902d2988e2b, entries=150, sequenceid=519, filesize=11.9 K 2023-04-03 13:58:13,242 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 899cf3d098e77c9ee3a377e83dee985e in 1269ms, sequenceid=519, compaction requested=true 2023-04-03 13:58:13,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:13,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:13,242 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-03 13:58:13,242 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-03 13:58:13,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:13,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:13,244 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 215716 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-03 13:58:13,244 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/A is initiating minor compaction (all files) 2023-04-03 13:58:13,244 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/A in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:13,244 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/99a6210209464c2ab3d71d2c9b3c8e07, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb3cb021f2334799ac72ed2e2ff3a968, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/2b483dde16634848bd895987a476db5a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3d6f6d47d53a4e4d82a303496dca57b1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9bf5cb7a4c0e4ba980f1e3adee8563f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb5d0b7a74d44593bf978ff465383246, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/261db5a298c744e1a9f98687e4fa69ce, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/194a52e73bbb4af7b592d2e375ab3669] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=210.7 K 2023-04-03 13:58:13,245 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 99a6210209464c2ab3d71d2c9b3c8e07, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680530283365 2023-04-03 13:58:13,245 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting cb3cb021f2334799ac72ed2e2ff3a968, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680530283395 2023-04-03 13:58:13,245 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2b483dde16634848bd895987a476db5a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680530284095 2023-04-03 13:58:13,245 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97796 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-03 13:58:13,245 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/B is initiating minor compaction (all files) 2023-04-03 13:58:13,245 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3d6f6d47d53a4e4d82a303496dca57b1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680530286250 2023-04-03 13:58:13,245 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/B in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:13,245 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e07642945df8421dba2d4ba619e343bc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d9e15eb01fcd4a71b895de4cf0de2b7f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/9ce476cce225447c981c39aa4b850239, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/65ec0324266c42fa85c86ccb98726b83, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/77fa0f432e434679b14c300ff40906bc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/86cee91554c14c50b0a27b280fec74ad, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/c2d71bfd92f84e21bd381b9b93f39b43, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/b14404e6291c4513853daeb46f69027d] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=95.5 K 2023-04-03 13:58:13,246 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9bf5cb7a4c0e4ba980f1e3adee8563f5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680530286882 2023-04-03 13:58:13,246 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e07642945df8421dba2d4ba619e343bc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680530283365 2023-04-03 13:58:13,246 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting cb5d0b7a74d44593bf978ff465383246, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680530287529 2023-04-03 13:58:13,246 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d9e15eb01fcd4a71b895de4cf0de2b7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680530283395 2023-04-03 13:58:13,246 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 261db5a298c744e1a9f98687e4fa69ce, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680530289683 2023-04-03 13:58:13,246 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 9ce476cce225447c981c39aa4b850239, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680530284095 2023-04-03 13:58:13,246 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 194a52e73bbb4af7b592d2e375ab3669, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680530291850 2023-04-03 13:58:13,247 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 65ec0324266c42fa85c86ccb98726b83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680530286250 2023-04-03 13:58:13,247 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 77fa0f432e434679b14c300ff40906bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680530286882 2023-04-03 13:58:13,248 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 86cee91554c14c50b0a27b280fec74ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680530287529 2023-04-03 13:58:13,248 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c2d71bfd92f84e21bd381b9b93f39b43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680530289683 2023-04-03 13:58:13,248 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting b14404e6291c4513853daeb46f69027d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680530291850 2023-04-03 13:58:13,268 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#B#compaction#346 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:13,273 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#A#compaction#345 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:13,279 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/d330b9c72d5b435492c665a3bcc511a9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d330b9c72d5b435492c665a3bcc511a9 2023-04-03 13:58:13,284 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/B of 899cf3d098e77c9ee3a377e83dee985e into d330b9c72d5b435492c665a3bcc511a9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:13,284 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:13,284 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/B, priority=8, startTime=1680530293242; duration=0sec 2023-04-03 13:58:13,284 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:13,284 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-03 13:58:13,285 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 97796 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-03 13:58:13,285 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/C is initiating minor compaction (all files) 2023-04-03 13:58:13,286 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/C in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:13,286 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0f25017f78484f7abec47e8d5cb0e76c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4f0131092cd84020aa4321ab43eaa896, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8df37611b60c4f5294db2f5516704ee8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/3ab82b17ee6a43a5b46f0967fe772739, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/379c12dd6ef54aa1aaf1ea54618a781c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/366b6dee003949f78a46fd5379c5346c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9a55283720134ddfad6d8ae4e6914cb5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0aa728f7de844b3ea5de4902d2988e2b] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=95.5 K 2023-04-03 13:58:13,287 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 0f25017f78484f7abec47e8d5cb0e76c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680530283365 2023-04-03 13:58:13,287 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4f0131092cd84020aa4321ab43eaa896, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680530283395 2023-04-03 13:58:13,287 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 8df37611b60c4f5294db2f5516704ee8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680530284095 2023-04-03 13:58:13,288 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 3ab82b17ee6a43a5b46f0967fe772739, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1680530286250 2023-04-03 13:58:13,288 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 379c12dd6ef54aa1aaf1ea54618a781c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=471, earliestPutTs=1680530286882 2023-04-03 13:58:13,288 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 366b6dee003949f78a46fd5379c5346c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680530287529 2023-04-03 13:58:13,288 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 9a55283720134ddfad6d8ae4e6914cb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680530289683 2023-04-03 13:58:13,288 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 0aa728f7de844b3ea5de4902d2988e2b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680530291850 2023-04-03 13:58:13,298 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#C#compaction#347 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:13,306 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/9f88928c4b5d4294bdfff225a51a6db8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9f88928c4b5d4294bdfff225a51a6db8 2023-04-03 13:58:13,310 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/C of 899cf3d098e77c9ee3a377e83dee985e into 9f88928c4b5d4294bdfff225a51a6db8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:13,310 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:13,310 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/C, priority=8, startTime=1680530293242; duration=0sec 2023-04-03 13:58:13,310 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:13,313 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:13,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:13,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:13,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:13,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:13,317 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:13,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:13,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:13,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:13,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:13,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:13,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:13,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:13,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:13,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:13,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:13,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:13,319 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:13,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:13,320 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 13:58:13,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:13,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:13,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:13,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:13,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:13,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:13,734 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403076df66431d94793be05b5fbbce46107 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403076df66431d94793be05b5fbbce46107 2023-04-03 13:58:14,096 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/093ff2521cfd4dd4a0123c64d7e4623e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/093ff2521cfd4dd4a0123c64d7e4623e 2023-04-03 13:58:14,099 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/A of 899cf3d098e77c9ee3a377e83dee985e into 093ff2521cfd4dd4a0123c64d7e4623e(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:14,099 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:14,099 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/A, priority=8, startTime=1680530293242; duration=0sec 2023-04-03 13:58:14,099 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:14,139 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=537, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/3c67bdd578be4649911ec34dbc33c3e8 2023-04-03 13:58:14,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:14,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:14,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530354163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530354164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530354165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530354166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530354269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530354269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530354269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530354270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530354473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530354473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530354473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530354473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,548 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/5aecfcc1e43b452cae72bdcf3fac1e80 2023-04-03 13:58:14,556 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/42258a6761784e9f96f173408a78e498 2023-04-03 13:58:14,559 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/3c67bdd578be4649911ec34dbc33c3e8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3c67bdd578be4649911ec34dbc33c3e8 2023-04-03 13:58:14,561 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3c67bdd578be4649911ec34dbc33c3e8, entries=150, sequenceid=537, filesize=25.4 K 2023-04-03 13:58:14,562 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/5aecfcc1e43b452cae72bdcf3fac1e80 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5aecfcc1e43b452cae72bdcf3fac1e80 2023-04-03 13:58:14,565 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5aecfcc1e43b452cae72bdcf3fac1e80, entries=150, sequenceid=537, filesize=11.9 K 2023-04-03 13:58:14,566 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/42258a6761784e9f96f173408a78e498 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/42258a6761784e9f96f173408a78e498 2023-04-03 13:58:14,568 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/42258a6761784e9f96f173408a78e498, entries=150, sequenceid=537, filesize=11.9 K 2023-04-03 13:58:14,569 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 899cf3d098e77c9ee3a377e83dee985e in 1249ms, sequenceid=537, compaction requested=false 2023-04-03 13:58:14,569 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:14,569 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:14,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:14,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:14,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:14,569 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:14,570 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:14,570 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:14,571 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:14,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:14,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88e55a7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:14,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:14,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:14,573 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:14,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:14,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:14,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:14,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:14,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:14,573 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,574 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:14,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:14,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:14,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:14,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:14,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:14,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:14,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:14,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:14,577 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:14,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:14,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:14,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:14,577 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:14,579 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,579 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,579 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:14,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:14,580 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:14,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,581 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:14,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:14,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:14,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:14,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:14,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:14,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:14,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:14,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:14,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:14,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:14,583 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:14,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,583 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:14,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:14,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:14,683 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:14,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:14,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:14,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:14,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:14,686 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:14,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:14,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:14,686 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:14,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:14,687 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:14,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:14,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:14,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:14,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:14,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:14,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:14,688 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:14,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:14,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:14,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:14,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:14,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:14,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. started... 2023-04-03 13:58:14,689 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 13:58:14,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:14,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:14,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:14,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:14,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:14,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:14,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:14,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. as already flushing 2023-04-03 13:58:14,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530354788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530354791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530354791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530354792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530354895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530354895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530354895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:14,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:14,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530354897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,099 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403317d4e584af44a309e00c846a3feef9e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403317d4e584af44a309e00c846a3feef9e 2023-04-03 13:58:15,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530355100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530355100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530355100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530355101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530355406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530355406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,410 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530355407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530355407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,503 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=554, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/fc2d780ec687438d8ffbbc6419da19b7 2023-04-03 13:58:15,913 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/5e62d096f9244d1ba17bc5d0f3402844 2023-04-03 13:58:15,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42084 deadline: 1680530355913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42096 deadline: 1680530355915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42110 deadline: 1680530355915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:15,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:15,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42066 deadline: 1680530355915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,218 DEBUG [Thread-2320] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52249391 to 127.0.0.1:57081 2023-04-03 13:58:16,218 DEBUG [Thread-2320] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:16,223 DEBUG [Thread-2318] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x060c5cba to 127.0.0.1:57081 2023-04-03 13:58:16,223 DEBUG [Thread-2318] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:16,224 DEBUG [Thread-2322] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ba3ce53 to 127.0.0.1:57081 2023-04-03 13:58:16,224 DEBUG [Thread-2322] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:16,224 DEBUG [Thread-2316] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74d6bf18 to 127.0.0.1:57081 2023-04-03 13:58:16,224 DEBUG [Thread-2316] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:16,229 DEBUG [Thread-2324] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5144fcff to 127.0.0.1:57081 2023-04-03 13:58:16,229 DEBUG [Thread-2324] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:16,296 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-03 13:58:16,321 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/e7b27332bd5c4a189b7d57eaa64a2cee, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/df86791455bd4d96bc4658e7071c7f51, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/95d04ded359847d6a10578a7baeca19b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/d8ba66011f5947fc94d1229f8f801265, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/475bba786b4e48a4905eb399a856c0f1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/68acc82de7e84c0d853ac3ebe035cec2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/5d4d2b27d8ad41298340cfbc83c46014, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3ee323f0b76943e89e60b4594a8c5225, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/316bc924d46342568cb3ae2ea3536c1e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/99a6210209464c2ab3d71d2c9b3c8e07, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb3cb021f2334799ac72ed2e2ff3a968, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/2b483dde16634848bd895987a476db5a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3d6f6d47d53a4e4d82a303496dca57b1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9bf5cb7a4c0e4ba980f1e3adee8563f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb5d0b7a74d44593bf978ff465383246, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/261db5a298c744e1a9f98687e4fa69ce, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/194a52e73bbb4af7b592d2e375ab3669] to archive 2023-04-03 13:58:16,322 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:16,322 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/8dcb293bbfdc457faa94d739b93346e1 2023-04-03 13:58:16,323 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/e7b27332bd5c4a189b7d57eaa64a2cee to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/e7b27332bd5c4a189b7d57eaa64a2cee 2023-04-03 13:58:16,324 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/df86791455bd4d96bc4658e7071c7f51 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/df86791455bd4d96bc4658e7071c7f51 2023-04-03 13:58:16,325 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/95d04ded359847d6a10578a7baeca19b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/95d04ded359847d6a10578a7baeca19b 2023-04-03 13:58:16,326 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/d8ba66011f5947fc94d1229f8f801265 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/d8ba66011f5947fc94d1229f8f801265 2023-04-03 13:58:16,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/fc2d780ec687438d8ffbbc6419da19b7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/fc2d780ec687438d8ffbbc6419da19b7 2023-04-03 13:58:16,327 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/475bba786b4e48a4905eb399a856c0f1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/475bba786b4e48a4905eb399a856c0f1 2023-04-03 13:58:16,327 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/68acc82de7e84c0d853ac3ebe035cec2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/68acc82de7e84c0d853ac3ebe035cec2 2023-04-03 13:58:16,328 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/5d4d2b27d8ad41298340cfbc83c46014 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/5d4d2b27d8ad41298340cfbc83c46014 2023-04-03 13:58:16,329 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3ee323f0b76943e89e60b4594a8c5225 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3ee323f0b76943e89e60b4594a8c5225 2023-04-03 13:58:16,330 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/316bc924d46342568cb3ae2ea3536c1e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/316bc924d46342568cb3ae2ea3536c1e 2023-04-03 13:58:16,331 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/fc2d780ec687438d8ffbbc6419da19b7, entries=150, sequenceid=554, filesize=25.4 K 2023-04-03 13:58:16,331 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/99a6210209464c2ab3d71d2c9b3c8e07 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/99a6210209464c2ab3d71d2c9b3c8e07 2023-04-03 13:58:16,331 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/5e62d096f9244d1ba17bc5d0f3402844 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5e62d096f9244d1ba17bc5d0f3402844 2023-04-03 13:58:16,332 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb3cb021f2334799ac72ed2e2ff3a968 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb3cb021f2334799ac72ed2e2ff3a968 2023-04-03 13:58:16,333 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/2b483dde16634848bd895987a476db5a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/2b483dde16634848bd895987a476db5a 2023-04-03 13:58:16,334 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3d6f6d47d53a4e4d82a303496dca57b1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3d6f6d47d53a4e4d82a303496dca57b1 2023-04-03 13:58:16,335 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9bf5cb7a4c0e4ba980f1e3adee8563f5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9bf5cb7a4c0e4ba980f1e3adee8563f5 2023-04-03 13:58:16,335 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5e62d096f9244d1ba17bc5d0f3402844, entries=150, sequenceid=554, filesize=11.9 K 2023-04-03 13:58:16,336 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/8dcb293bbfdc457faa94d739b93346e1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8dcb293bbfdc457faa94d739b93346e1 2023-04-03 13:58:16,336 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb5d0b7a74d44593bf978ff465383246 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/cb5d0b7a74d44593bf978ff465383246 2023-04-03 13:58:16,337 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/261db5a298c744e1a9f98687e4fa69ce to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/261db5a298c744e1a9f98687e4fa69ce 2023-04-03 13:58:16,338 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/194a52e73bbb4af7b592d2e375ab3669 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/194a52e73bbb4af7b592d2e375ab3669 2023-04-03 13:58:16,342 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8dcb293bbfdc457faa94d739b93346e1, entries=150, sequenceid=554, filesize=11.9 K 2023-04-03 13:58:16,342 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 899cf3d098e77c9ee3a377e83dee985e in 1653ms, sequenceid=554, compaction requested=true 2023-04-03 13:58:16,343 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:16,343 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:16,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:16,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:16,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:16,351 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:16,353 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:16,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:16,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:16,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:16,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:16,354 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1288f737b59644ce849f8b43cd407176, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3f72d0231a524062a033e1979b347469, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/fa57f645fd7d4dcb9ec8d212ca518210, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/ea371f743761460fbaccdecc690265d7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3add8ba5b55f4e1887440d616e344ec6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/cf32d7490af9472881dcf0586557007b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/407ae9a76e8e41fb8cb14d43bbfff94c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/6dabae4e2293453881bfe7388b715f3f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e07642945df8421dba2d4ba619e343bc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/738ba42aa5fd43c7a299594da668bb6c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d9e15eb01fcd4a71b895de4cf0de2b7f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/9ce476cce225447c981c39aa4b850239, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/65ec0324266c42fa85c86ccb98726b83, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/77fa0f432e434679b14c300ff40906bc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/86cee91554c14c50b0a27b280fec74ad, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/c2d71bfd92f84e21bd381b9b93f39b43, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/b14404e6291c4513853daeb46f69027d] to archive 2023-04-03 13:58:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,354 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:16,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:16,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:16,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d26519f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:16,355 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:16,355 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:16,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:16,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:16,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:16,356 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:16,356 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:16,356 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1288f737b59644ce849f8b43cd407176 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1288f737b59644ce849f8b43cd407176 2023-04-03 13:58:16,356 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:16,357 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3f72d0231a524062a033e1979b347469 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3f72d0231a524062a033e1979b347469 2023-04-03 13:58:16,358 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/fa57f645fd7d4dcb9ec8d212ca518210 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/fa57f645fd7d4dcb9ec8d212ca518210 2023-04-03 13:58:16,360 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/ea371f743761460fbaccdecc690265d7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/ea371f743761460fbaccdecc690265d7 2023-04-03 13:58:16,361 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3add8ba5b55f4e1887440d616e344ec6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/3add8ba5b55f4e1887440d616e344ec6 2023-04-03 13:58:16,362 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/cf32d7490af9472881dcf0586557007b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/cf32d7490af9472881dcf0586557007b 2023-04-03 13:58:16,362 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,362 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:16,363 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:16,363 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:16,363 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,363 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:16,363 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/407ae9a76e8e41fb8cb14d43bbfff94c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/407ae9a76e8e41fb8cb14d43bbfff94c 2023-04-03 13:58:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,364 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/6dabae4e2293453881bfe7388b715f3f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/6dabae4e2293453881bfe7388b715f3f 2023-04-03 13:58:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,366 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e07642945df8421dba2d4ba619e343bc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e07642945df8421dba2d4ba619e343bc 2023-04-03 13:58:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:16,366 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:16,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:16,366 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:16,366 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:16,366 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:16,366 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:16,366 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/738ba42aa5fd43c7a299594da668bb6c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/738ba42aa5fd43c7a299594da668bb6c 2023-04-03 13:58:16,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:16,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:16,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:16,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:16,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:16,367 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d9e15eb01fcd4a71b895de4cf0de2b7f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d9e15eb01fcd4a71b895de4cf0de2b7f 2023-04-03 13:58:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:16,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:16,368 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/9ce476cce225447c981c39aa4b850239 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/9ce476cce225447c981c39aa4b850239 2023-04-03 13:58:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,368 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,369 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/65ec0324266c42fa85c86ccb98726b83 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/65ec0324266c42fa85c86ccb98726b83 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:16,370 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:16,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:16,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:16,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:16,370 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/77fa0f432e434679b14c300ff40906bc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/77fa0f432e434679b14c300ff40906bc 2023-04-03 13:58:16,370 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:16,370 DEBUG [Thread-2315] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:16,370 DEBUG [Thread-2315] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:16,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:16,371 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/86cee91554c14c50b0a27b280fec74ad to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/86cee91554c14c50b0a27b280fec74ad 2023-04-03 13:58:16,372 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/c2d71bfd92f84e21bd381b9b93f39b43 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/c2d71bfd92f84e21bd381b9b93f39b43 2023-04-03 13:58:16,372 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/b14404e6291c4513853daeb46f69027d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/b14404e6291c4513853daeb46f69027d 2023-04-03 13:58:16,375 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fa0f7134f8494d948186c9e3748afc5a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/6d5fac33ac494e26bcdc74ed53b967d2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/e5782ab1a0d4407da6b958b34738462b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fb7869da1d36402aa2ac8f0f6179472f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5b68078a5c434d5f83b748be28480e3d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/aafe5fff7c42463fbd626af73024c5d9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8179f471c3504bc2a7a8f585343aab32, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/bea9cf9ccf2b40e2a91b79a37c97b609, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0f25017f78484f7abec47e8d5cb0e76c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5df93dafdf764c708044230c1324be40, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4f0131092cd84020aa4321ab43eaa896, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8df37611b60c4f5294db2f5516704ee8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/3ab82b17ee6a43a5b46f0967fe772739, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/379c12dd6ef54aa1aaf1ea54618a781c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/366b6dee003949f78a46fd5379c5346c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9a55283720134ddfad6d8ae4e6914cb5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0aa728f7de844b3ea5de4902d2988e2b] to archive 2023-04-03 13:58:16,376 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:16,377 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fa0f7134f8494d948186c9e3748afc5a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fa0f7134f8494d948186c9e3748afc5a 2023-04-03 13:58:16,378 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/6d5fac33ac494e26bcdc74ed53b967d2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/6d5fac33ac494e26bcdc74ed53b967d2 2023-04-03 13:58:16,379 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/e5782ab1a0d4407da6b958b34738462b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/e5782ab1a0d4407da6b958b34738462b 2023-04-03 13:58:16,380 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fb7869da1d36402aa2ac8f0f6179472f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/fb7869da1d36402aa2ac8f0f6179472f 2023-04-03 13:58:16,381 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5b68078a5c434d5f83b748be28480e3d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5b68078a5c434d5f83b748be28480e3d 2023-04-03 13:58:16,382 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/aafe5fff7c42463fbd626af73024c5d9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/aafe5fff7c42463fbd626af73024c5d9 2023-04-03 13:58:16,383 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8179f471c3504bc2a7a8f585343aab32 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8179f471c3504bc2a7a8f585343aab32 2023-04-03 13:58:16,384 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/bea9cf9ccf2b40e2a91b79a37c97b609 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/bea9cf9ccf2b40e2a91b79a37c97b609 2023-04-03 13:58:16,385 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0f25017f78484f7abec47e8d5cb0e76c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0f25017f78484f7abec47e8d5cb0e76c 2023-04-03 13:58:16,386 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5df93dafdf764c708044230c1324be40 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/5df93dafdf764c708044230c1324be40 2023-04-03 13:58:16,387 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4f0131092cd84020aa4321ab43eaa896 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4f0131092cd84020aa4321ab43eaa896 2023-04-03 13:58:16,388 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8df37611b60c4f5294db2f5516704ee8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8df37611b60c4f5294db2f5516704ee8 2023-04-03 13:58:16,389 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/3ab82b17ee6a43a5b46f0967fe772739 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/3ab82b17ee6a43a5b46f0967fe772739 2023-04-03 13:58:16,390 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/379c12dd6ef54aa1aaf1ea54618a781c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/379c12dd6ef54aa1aaf1ea54618a781c 2023-04-03 13:58:16,391 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/366b6dee003949f78a46fd5379c5346c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/366b6dee003949f78a46fd5379c5346c 2023-04-03 13:58:16,392 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9a55283720134ddfad6d8ae4e6914cb5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9a55283720134ddfad6d8ae4e6914cb5 2023-04-03 13:58:16,393 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0aa728f7de844b3ea5de4902d2988e2b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/0aa728f7de844b3ea5de4902d2988e2b 2023-04-03 13:58:16,470 DEBUG [Thread-2315] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:16,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:16,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:16,929 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 13:58:16,929 DEBUG [Thread-2307] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x377ba302 to 127.0.0.1:57081 2023-04-03 13:58:16,929 DEBUG [Thread-2305] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ba43635 to 127.0.0.1:57081 2023-04-03 13:58:16,929 DEBUG [Thread-2307] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:16,929 DEBUG [Thread-2305] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:16,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:16,931 DEBUG [Thread-2311] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d63a8ce to 127.0.0.1:57081 2023-04-03 13:58:16,931 DEBUG [Thread-2311] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:16,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:16,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:16,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:16,931 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:16,931 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:16,932 DEBUG [Thread-2313] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d80e5a8 to 127.0.0.1:57081 2023-04-03 13:58:16,932 DEBUG [Thread-2313] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:17,349 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040324931f641a86433c91228a914417fe8e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040324931f641a86433c91228a914417fe8e 2023-04-03 13:58:17,756 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=573, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/750d6f1ab7ab46f39328f19b99177079 2023-04-03 13:58:18,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/939218a5c00d4fca92d2ad77c32f2adf 2023-04-03 13:58:18,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/428f7c76b8204d71b5463287216a44ba 2023-04-03 13:58:18,599 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/750d6f1ab7ab46f39328f19b99177079 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/750d6f1ab7ab46f39328f19b99177079 2023-04-03 13:58:18,606 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/750d6f1ab7ab46f39328f19b99177079, entries=150, sequenceid=573, filesize=25.4 K 2023-04-03 13:58:18,607 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/939218a5c00d4fca92d2ad77c32f2adf as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/939218a5c00d4fca92d2ad77c32f2adf 2023-04-03 13:58:18,613 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/939218a5c00d4fca92d2ad77c32f2adf, entries=150, sequenceid=573, filesize=11.9 K 2023-04-03 13:58:18,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/428f7c76b8204d71b5463287216a44ba as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/428f7c76b8204d71b5463287216a44ba 2023-04-03 13:58:18,619 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/428f7c76b8204d71b5463287216a44ba, entries=150, sequenceid=573, filesize=11.9 K 2023-04-03 13:58:18,621 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=6.71 KB/6870 for 899cf3d098e77c9ee3a377e83dee985e in 1692ms, sequenceid=573, compaction requested=true 2023-04-03 13:58:18,621 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:18,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:18,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:18,621 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:18,621 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:18,621 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:18,623 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104764 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:18,623 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/A is initiating minor compaction (all files) 2023-04-03 13:58:18,623 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:18,623 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/A in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:18,623 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/B is initiating minor compaction (all files) 2023-04-03 13:58:18,623 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/093ff2521cfd4dd4a0123c64d7e4623e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3c67bdd578be4649911ec34dbc33c3e8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/fc2d780ec687438d8ffbbc6419da19b7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/750d6f1ab7ab46f39328f19b99177079] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=102.3 K 2023-04-03 13:58:18,624 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/B in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:18,624 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d330b9c72d5b435492c665a3bcc511a9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5aecfcc1e43b452cae72bdcf3fac1e80, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5e62d096f9244d1ba17bc5d0f3402844, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/939218a5c00d4fca92d2ad77c32f2adf] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=48.2 K 2023-04-03 13:58:18,624 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 093ff2521cfd4dd4a0123c64d7e4623e, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680530291850 2023-04-03 13:58:18,624 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d330b9c72d5b435492c665a3bcc511a9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680530291850 2023-04-03 13:58:18,624 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3c67bdd578be4649911ec34dbc33c3e8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530291985 2023-04-03 13:58:18,624 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5aecfcc1e43b452cae72bdcf3fac1e80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530291985 2023-04-03 13:58:18,625 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting fc2d780ec687438d8ffbbc6419da19b7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1680530294161 2023-04-03 13:58:18,625 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5e62d096f9244d1ba17bc5d0f3402844, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1680530294161 2023-04-03 13:58:18,625 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 750d6f1ab7ab46f39328f19b99177079, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680530294791 2023-04-03 13:58:18,625 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 939218a5c00d4fca92d2ad77c32f2adf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680530294791 2023-04-03 13:58:18,633 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#B#compaction#357 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:18,638 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#A#compaction#358 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:18,642 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/1b9d7cfbf53e40ed8d7dec99555c3a56 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1b9d7cfbf53e40ed8d7dec99555c3a56 2023-04-03 13:58:18,646 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/B of 899cf3d098e77c9ee3a377e83dee985e into 1b9d7cfbf53e40ed8d7dec99555c3a56(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:18,646 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:18,646 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/B, priority=12, startTime=1680530298621; duration=0sec 2023-04-03 13:58:18,646 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:18,647 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:18,648 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:18,648 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): 899cf3d098e77c9ee3a377e83dee985e/C is initiating minor compaction (all files) 2023-04-03 13:58:18,648 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 899cf3d098e77c9ee3a377e83dee985e/C in TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:18,648 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9f88928c4b5d4294bdfff225a51a6db8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/42258a6761784e9f96f173408a78e498, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8dcb293bbfdc457faa94d739b93346e1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/428f7c76b8204d71b5463287216a44ba] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp, totalSize=48.2 K 2023-04-03 13:58:18,648 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 9f88928c4b5d4294bdfff225a51a6db8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680530291850 2023-04-03 13:58:18,648 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 42258a6761784e9f96f173408a78e498, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530291985 2023-04-03 13:58:18,648 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 8dcb293bbfdc457faa94d739b93346e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1680530294161 2023-04-03 13:58:18,649 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 428f7c76b8204d71b5463287216a44ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1680530294791 2023-04-03 13:58:18,654 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): 899cf3d098e77c9ee3a377e83dee985e#C#compaction#359 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:19,066 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/4fac12d0aea04bd7a4d4723c92bf58db as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4fac12d0aea04bd7a4d4723c92bf58db 2023-04-03 13:58:19,072 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/C of 899cf3d098e77c9ee3a377e83dee985e into 4fac12d0aea04bd7a4d4723c92bf58db(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:19,072 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:19,072 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/C, priority=12, startTime=1680530298621; duration=0sec 2023-04-03 13:58:19,072 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:19,464 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/b78042b4109542648eb804160be05466 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/b78042b4109542648eb804160be05466 2023-04-03 13:58:19,468 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 899cf3d098e77c9ee3a377e83dee985e/A of 899cf3d098e77c9ee3a377e83dee985e into b78042b4109542648eb804160be05466(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:19,468 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:19,468 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e., storeName=899cf3d098e77c9ee3a377e83dee985e/A, priority=12, startTime=1680530298621; duration=0sec 2023-04-03 13:58:19,468 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:22,334 DEBUG [Thread-2309] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1eaa86ba to 127.0.0.1:57081 2023-04-03 13:58:22,335 DEBUG [Thread-2309] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:22,335 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 13:58:22,335 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-04-03 13:58:22,335 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-03 13:58:22,336 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 133 2023-04-03 13:58:22,336 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-04-03 13:58:22,336 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 145 2023-04-03 13:58:22,336 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 13:58:22,336 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 13:58:22,336 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 1767 2023-04-03 13:58:22,336 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 5298 rows 2023-04-03 13:58:22,336 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 1764 2023-04-03 13:58:22,337 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 5288 rows 2023-04-03 13:58:22,337 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 1777 2023-04-03 13:58:22,337 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 5330 rows 2023-04-03 13:58:22,337 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 1764 2023-04-03 13:58:22,337 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 5287 rows 2023-04-03 13:58:22,337 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 1777 2023-04-03 13:58:22,337 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 5331 rows 2023-04-03 13:58:22,338 INFO [Listener at localhost.localdomain/43477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 13:58:22,338 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fa3f5c5 to 127.0.0.1:57081 2023-04-03 13:58:22,338 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:22,340 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 13:58:22,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-03 13:58:22,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:22,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 13:58:22,344 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530302344"}]},"ts":"1680530302344"} 2023-04-03 13:58:22,345 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 13:58:22,443 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 13:58:22,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, UNASSIGN}] 2023-04-03 13:58:22,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 13:58:22,447 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, UNASSIGN 2023-04-03 13:58:22,448 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=899cf3d098e77c9ee3a377e83dee985e, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:22,448 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530302448"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530302448"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530302448"}]},"ts":"1680530302448"} 2023-04-03 13:58:22,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:58:22,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:22,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 899cf3d098e77c9ee3a377e83dee985e, disabling compactions & flushes 2023-04-03 13:58:22,604 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:22,604 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:22,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. after waiting 0 ms 2023-04-03 13:58:22,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:22,605 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 899cf3d098e77c9ee3a377e83dee985e 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-03 13:58:22,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=A 2023-04-03 13:58:22,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:22,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=B 2023-04-03 13:58:22,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:22,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 899cf3d098e77c9ee3a377e83dee985e, store=C 2023-04-03 13:58:22,605 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:22,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 13:58:22,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 13:58:23,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a5bb72e5f0544a89a2b85a95b292153c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a5bb72e5f0544a89a2b85a95b292153c 2023-04-03 13:58:23,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=581, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/9179cb184db941ce9d02052e7e99ddfe 2023-04-03 13:58:23,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/e7ad3fb059054d0487224a197cf52ef8 2023-04-03 13:58:23,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 13:58:23,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/b166fc104ab64edc925437759c1c8159 2023-04-03 13:58:23,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/A/9179cb184db941ce9d02052e7e99ddfe as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9179cb184db941ce9d02052e7e99ddfe 2023-04-03 13:58:23,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9179cb184db941ce9d02052e7e99ddfe, entries=50, sequenceid=581, filesize=11.7 K 2023-04-03 13:58:23,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/B/e7ad3fb059054d0487224a197cf52ef8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e7ad3fb059054d0487224a197cf52ef8 2023-04-03 13:58:23,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e7ad3fb059054d0487224a197cf52ef8, entries=50, sequenceid=581, filesize=7.1 K 2023-04-03 13:58:23,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/.tmp/C/b166fc104ab64edc925437759c1c8159 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/b166fc104ab64edc925437759c1c8159 2023-04-03 13:58:23,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/b166fc104ab64edc925437759c1c8159, entries=50, sequenceid=581, filesize=7.1 K 2023-04-03 13:58:23,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 899cf3d098e77c9ee3a377e83dee985e in 1272ms, sequenceid=581, compaction requested=false 2023-04-03 13:58:23,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/093ff2521cfd4dd4a0123c64d7e4623e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3c67bdd578be4649911ec34dbc33c3e8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/fc2d780ec687438d8ffbbc6419da19b7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/750d6f1ab7ab46f39328f19b99177079] to archive 2023-04-03 13:58:23,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:23,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/093ff2521cfd4dd4a0123c64d7e4623e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/093ff2521cfd4dd4a0123c64d7e4623e 2023-04-03 13:58:23,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3c67bdd578be4649911ec34dbc33c3e8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/3c67bdd578be4649911ec34dbc33c3e8 2023-04-03 13:58:23,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/fc2d780ec687438d8ffbbc6419da19b7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/fc2d780ec687438d8ffbbc6419da19b7 2023-04-03 13:58:23,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/750d6f1ab7ab46f39328f19b99177079 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/750d6f1ab7ab46f39328f19b99177079 2023-04-03 13:58:23,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d330b9c72d5b435492c665a3bcc511a9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5aecfcc1e43b452cae72bdcf3fac1e80, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5e62d096f9244d1ba17bc5d0f3402844, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/939218a5c00d4fca92d2ad77c32f2adf] to archive 2023-04-03 13:58:23,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:23,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d330b9c72d5b435492c665a3bcc511a9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/d330b9c72d5b435492c665a3bcc511a9 2023-04-03 13:58:23,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5aecfcc1e43b452cae72bdcf3fac1e80 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5aecfcc1e43b452cae72bdcf3fac1e80 2023-04-03 13:58:23,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5e62d096f9244d1ba17bc5d0f3402844 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/5e62d096f9244d1ba17bc5d0f3402844 2023-04-03 13:58:23,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/939218a5c00d4fca92d2ad77c32f2adf to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/939218a5c00d4fca92d2ad77c32f2adf 2023-04-03 13:58:23,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9f88928c4b5d4294bdfff225a51a6db8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/42258a6761784e9f96f173408a78e498, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8dcb293bbfdc457faa94d739b93346e1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/428f7c76b8204d71b5463287216a44ba] to archive 2023-04-03 13:58:23,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:23,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9f88928c4b5d4294bdfff225a51a6db8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/9f88928c4b5d4294bdfff225a51a6db8 2023-04-03 13:58:23,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/42258a6761784e9f96f173408a78e498 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/42258a6761784e9f96f173408a78e498 2023-04-03 13:58:23,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8dcb293bbfdc457faa94d739b93346e1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/8dcb293bbfdc457faa94d739b93346e1 2023-04-03 13:58:23,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/428f7c76b8204d71b5463287216a44ba to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/428f7c76b8204d71b5463287216a44ba 2023-04-03 13:58:23,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/recovered.edits/584.seqid, newMaxSeqId=584, maxSeqId=4 2023-04-03 13:58:23,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e. 2023-04-03 13:58:23,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 899cf3d098e77c9ee3a377e83dee985e: 2023-04-03 13:58:23,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:23,900 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=899cf3d098e77c9ee3a377e83dee985e, regionState=CLOSED 2023-04-03 13:58:23,900 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530303900"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530303900"}]},"ts":"1680530303900"} 2023-04-03 13:58:23,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-03 13:58:23,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 899cf3d098e77c9ee3a377e83dee985e, server=jenkins-hbase20.apache.org,39695,1680530174856 in 1.4500 sec 2023-04-03 13:58:23,903 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-03 13:58:23,903 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=899cf3d098e77c9ee3a377e83dee985e, UNASSIGN in 1.4570 sec 2023-04-03 13:58:23,903 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530303903"}]},"ts":"1680530303903"} 2023-04-03 13:58:23,904 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 13:58:23,905 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 13:58:23,906 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5640 sec 2023-04-03 13:58:24,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 13:58:24,452 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-03 13:58:24,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-03 13:58:24,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:24,457 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:24,458 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:24,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-03 13:58:24,467 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:24,471 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/recovered.edits] 2023-04-03 13:58:24,476 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9179cb184db941ce9d02052e7e99ddfe to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/9179cb184db941ce9d02052e7e99ddfe 2023-04-03 13:58:24,479 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/b78042b4109542648eb804160be05466 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/A/b78042b4109542648eb804160be05466 2023-04-03 13:58:24,483 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1b9d7cfbf53e40ed8d7dec99555c3a56 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/1b9d7cfbf53e40ed8d7dec99555c3a56 2023-04-03 13:58:24,485 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e7ad3fb059054d0487224a197cf52ef8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/B/e7ad3fb059054d0487224a197cf52ef8 2023-04-03 13:58:24,488 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4fac12d0aea04bd7a4d4723c92bf58db to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/4fac12d0aea04bd7a4d4723c92bf58db 2023-04-03 13:58:24,489 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/b166fc104ab64edc925437759c1c8159 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/C/b166fc104ab64edc925437759c1c8159 2023-04-03 13:58:24,492 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/recovered.edits/584.seqid to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e/recovered.edits/584.seqid 2023-04-03 13:58:24,493 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/899cf3d098e77c9ee3a377e83dee985e 2023-04-03 13:58:24,493 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:58:24,493 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 13:58:24,494 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 13:58:24,497 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403076df66431d94793be05b5fbbce46107 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403076df66431d94793be05b5fbbce46107 2023-04-03 13:58:24,498 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040307e780594abf4105875eb852492a8a20 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040307e780594abf4105875eb852492a8a20 2023-04-03 13:58:24,500 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403118f88ab37ae46fc8ba3879c11dab171 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403118f88ab37ae46fc8ba3879c11dab171 2023-04-03 13:58:24,501 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040318a4be909a5c4b6b942060cdd6c5c5e5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040318a4be909a5c4b6b942060cdd6c5c5e5 2023-04-03 13:58:24,502 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040324931f641a86433c91228a914417fe8e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040324931f641a86433c91228a914417fe8e 2023-04-03 13:58:24,503 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403317d4e584af44a309e00c846a3feef9e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403317d4e584af44a309e00c846a3feef9e 2023-04-03 13:58:24,504 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034c94a403542c498694b17ea9444b80c0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034c94a403542c498694b17ea9444b80c0 2023-04-03 13:58:24,505 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040357b7f16d862b4585be7833e284f7e8ce to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040357b7f16d862b4585be7833e284f7e8ce 2023-04-03 13:58:24,505 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040359a39df43d9f4efc85126e8bc72b0128 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040359a39df43d9f4efc85126e8bc72b0128 2023-04-03 13:58:24,507 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036c9734587ff54740b92cd2f31603c6b1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036c9734587ff54740b92cd2f31603c6b1 2023-04-03 13:58:24,508 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039464b3b507384b5f969254bc4884adfc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039464b3b507384b5f969254bc4884adfc 2023-04-03 13:58:24,508 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a5bb72e5f0544a89a2b85a95b292153c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a5bb72e5f0544a89a2b85a95b292153c 2023-04-03 13:58:24,509 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403af8b449f235f4ae8ac3a608f3616fa36 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403af8b449f235f4ae8ac3a608f3616fa36 2023-04-03 13:58:24,510 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c94e14d2267147b78384c89dbf976015 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c94e14d2267147b78384c89dbf976015 2023-04-03 13:58:24,511 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d154bdcb78cc45b895b12e2cd1716543 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d154bdcb78cc45b895b12e2cd1716543 2023-04-03 13:58:24,511 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d3000feaa05941a488cc4c78a5979b8c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d3000feaa05941a488cc4c78a5979b8c 2023-04-03 13:58:24,512 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e7d656d210aa43a0b96d55d8488b604e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e7d656d210aa43a0b96d55d8488b604e 2023-04-03 13:58:24,513 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e84ecdda92304e4d8477b14017ed624c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e84ecdda92304e4d8477b14017ed624c 2023-04-03 13:58:24,514 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fb379fc7de5843d7b63d46301e52cb58 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fb379fc7de5843d7b63d46301e52cb58 2023-04-03 13:58:24,514 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 13:58:24,516 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:24,518 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 13:58:24,520 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 13:58:24,521 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:24,521 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 13:58:24,521 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680530304521"}]},"ts":"9223372036854775807"} 2023-04-03 13:58:24,523 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 13:58:24,523 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 899cf3d098e77c9ee3a377e83dee985e, NAME => 'TestAcidGuarantees,,1680530274072.899cf3d098e77c9ee3a377e83dee985e.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 13:58:24,523 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 13:58:24,523 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680530304523"}]},"ts":"9223372036854775807"} 2023-04-03 13:58:24,527 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 13:58:24,528 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:24,529 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 74 msec 2023-04-03 13:58:24,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-03 13:58:24,560 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-03 13:58:24,571 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 229), OpenFileDescriptor=519 (was 548), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=360 (was 362), ProcessCount=166 (was 166), AvailableMemoryMB=3514 (was 3852) 2023-04-03 13:58:24,581 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=519, MaxFileDescriptor=60000, SystemLoadAverage=360, ProcessCount=166, AvailableMemoryMB=3514 2023-04-03 13:58:24,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:58:24,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:58:24,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:24,585 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 13:58:24,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-03 13:58:24,586 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 13:58:24,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 13:58:24,587 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:24,587 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8 empty. 2023-04-03 13:58:24,587 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:24,588 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:58:24,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 13:58:24,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 13:58:24,997 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:58:24,999 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a15553c96f61b5a8cc6d1615507e7af8, NAME => 'TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp 2023-04-03 13:58:25,021 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:58:25,021 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a15553c96f61b5a8cc6d1615507e7af8, disabling compactions & flushes 2023-04-03 13:58:25,021 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,021 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,021 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. after waiting 0 ms 2023-04-03 13:58:25,021 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,021 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,021 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:25,023 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 13:58:25,024 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530305023"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530305023"}]},"ts":"1680530305023"} 2023-04-03 13:58:25,024 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 13:58:25,025 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 13:58:25,025 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530305025"}]},"ts":"1680530305025"} 2023-04-03 13:58:25,026 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 13:58:25,027 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a15553c96f61b5a8cc6d1615507e7af8, ASSIGN}] 2023-04-03 13:58:25,029 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a15553c96f61b5a8cc6d1615507e7af8, ASSIGN 2023-04-03 13:58:25,030 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a15553c96f61b5a8cc6d1615507e7af8, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=false 2023-04-03 13:58:25,182 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=a15553c96f61b5a8cc6d1615507e7af8, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:25,182 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530305182"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530305182"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530305182"}]},"ts":"1680530305182"} 2023-04-03 13:58:25,185 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:58:25,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 13:58:25,344 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,344 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a15553c96f61b5a8cc6d1615507e7af8, NAME => 'TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:58:25,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:58:25,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,348 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,350 DEBUG [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A 2023-04-03 13:58:25,350 DEBUG [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A 2023-04-03 13:58:25,350 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:25,350 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a15553c96f61b5a8cc6d1615507e7af8 columnFamilyName A 2023-04-03 13:58:25,351 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.HStore(310): Store=a15553c96f61b5a8cc6d1615507e7af8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:25,351 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,353 DEBUG [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B 2023-04-03 13:58:25,353 DEBUG [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B 2023-04-03 13:58:25,353 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:25,353 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a15553c96f61b5a8cc6d1615507e7af8 columnFamilyName B 2023-04-03 13:58:25,354 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.HStore(310): Store=a15553c96f61b5a8cc6d1615507e7af8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:25,354 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,355 DEBUG [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C 2023-04-03 13:58:25,356 DEBUG [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C 2023-04-03 13:58:25,356 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:25,356 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a15553c96f61b5a8cc6d1615507e7af8 columnFamilyName C 2023-04-03 13:58:25,356 INFO [StoreOpener-a15553c96f61b5a8cc6d1615507e7af8-1] regionserver.HStore(310): Store=a15553c96f61b5a8cc6d1615507e7af8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:25,357 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,357 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,360 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:58:25,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:58:25,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened a15553c96f61b5a8cc6d1615507e7af8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58833432, jitterRate=-0.1233135461807251}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:58:25,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:25,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., pid=49, masterSystemTime=1680530305337 2023-04-03 13:58:25,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,368 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=a15553c96f61b5a8cc6d1615507e7af8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:25,368 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530305368"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530305368"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530305368"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530305368"}]},"ts":"1680530305368"} 2023-04-03 13:58:25,372 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-03 13:58:25,372 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 in 186 msec 2023-04-03 13:58:25,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-03 13:58:25,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a15553c96f61b5a8cc6d1615507e7af8, ASSIGN in 345 msec 2023-04-03 13:58:25,375 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 13:58:25,375 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530305375"}]},"ts":"1680530305375"} 2023-04-03 13:58:25,376 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 13:58:25,378 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 13:58:25,379 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 795 msec 2023-04-03 13:58:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 13:58:25,693 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-03 13:58:25,694 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x517cd407 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,704 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13d9e4ff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,705 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,708 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40524, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,710 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 13:58:25,712 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:46378, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 13:58:25,714 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x3e895a5d to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,718 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9e26d67, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,718 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x6ea7ac79 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,722 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a885842, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,722 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x51ff7cd4 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,727 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e8c6f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,728 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x06846250 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,732 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2cb7a788, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,732 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x647f95f6 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,736 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41ccedcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,736 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x40e6f624 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,740 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6662e946, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,740 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5d7fcf to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,744 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d7deb76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,744 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x4ea9d51a to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,748 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f4b7807, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,749 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x31fd1c1f to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,760 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31fc0e3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,760 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x209faa3c to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:25,765 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79db57d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:25,772 DEBUG [hconnection-0x4a43faf2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,772 DEBUG [hconnection-0x4919ecd3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,773 DEBUG [hconnection-0x17c30646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,773 DEBUG [hconnection-0x2466d41f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,775 DEBUG [hconnection-0x52bf5e33-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,775 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40534, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,776 DEBUG [hconnection-0x6bbcafb8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:25,778 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40550, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,778 DEBUG [hconnection-0x57a40f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,780 DEBUG [hconnection-0x162f9832-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,780 DEBUG [hconnection-0x5d0342cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,784 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40574, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,784 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,784 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40556, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,782 DEBUG [hconnection-0x8406173-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:25,788 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40578, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,788 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40580, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:25,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:25,791 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:25,791 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40582, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,791 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40596, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,791 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:25,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:25,798 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:25,800 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:40608, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:25,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:25,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:25,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:25,805 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:25,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:25,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:25,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:25,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:25,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:25,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:25,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:25,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:25,807 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:25,807 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:25,807 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:25,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:25,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:25,809 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:25,809 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:25,810 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-03 13:58:25,810 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:25,810 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:25,811 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:25,811 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:25,811 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:25,811 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:25,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:25,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:25,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680530305821, max timestamp=1680530305826Num uniques 859; 2023-04-03 13:58:25,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 13:58:25,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680530305821, max timestamp=1680530305826Num uniques 859; 2023-04-03 13:58:25,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680530305821, max timestamp=1680530305826Num uniques 859; 2023-04-03 13:58:25,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 13:58:25,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 13:58:25,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:25,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530365837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530305835, max timestamp=1680530305841Num uniques 1118; 2023-04-03 13:58:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530305835, max timestamp=1680530305841Num uniques 1118; 2023-04-03 13:58:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-03 13:58:25,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-03 13:58:25,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530305835, max timestamp=1680530305841Num uniques 1118; 2023-04-03 13:58:25,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-03 13:58:25,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530305859, max timestamp=1680530305865Num uniques 1077; 2023-04-03 13:58:25,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:25,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:25,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530305859, max timestamp=1680530305865Num uniques 1077; 2023-04-03 13:58:25,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530365866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:25,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680530305859, max timestamp=1680530305865Num uniques 1077; 2023-04-03 13:58:25,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:25,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 13:58:25,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530305879, max timestamp=1680530305894Num uniques 1036; 2023-04-03 13:58:25,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530305879, max timestamp=1680530305894Num uniques 1036; 2023-04-03 13:58:25,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-03 13:58:25,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680530305879, max timestamp=1680530305894Num uniques 1036; 2023-04-03 13:58:25,896 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-03 13:58:25,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-03 13:58:25,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680530305937, max timestamp=1680530305945Num uniques 1045; 2023-04-03 13:58:25,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680530305937, max timestamp=1680530305945Num uniques 1045; 2023-04-03 13:58:25,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-03 13:58:25,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-03 13:58:25,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1680530305937, max timestamp=1680530305945Num uniques 1045; 2023-04-03 13:58:25,948 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-03 13:58:25,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:25,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530365986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:25,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:25,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530365990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:25,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1680530305954, max timestamp=1680530305992Num uniques 1154; 2023-04-03 13:58:25,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-03 13:58:25,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1680530305954, max timestamp=1680530305992Num uniques 1154; 2023-04-03 13:58:25,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1680530305954, max timestamp=1680530305992Num uniques 1154; 2023-04-03 13:58:25,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-03 13:58:25,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=13920, suffix heap size=147984,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=304 2023-04-03 13:58:26,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530366019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530366025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12035, totalHeapSize=27592, min timestamp=1680530306001, max timestamp=1680530306023Num uniques 1163; 2023-04-03 13:58:26,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=53255, new segment data size=12035, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-04-03 13:58:26,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12036, totalHeapSize=27592, min timestamp=1680530306001, max timestamp=1680530306023Num uniques 1163; 2023-04-03 13:58:26,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=53256, new segment data size=12036, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-04-03 13:58:26,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530306007, max timestamp=1680530306023Num uniques 1163; 2023-04-03 13:58:26,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=6870, suffix heap size=148920,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=150 2023-04-03 13:58:26,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680530306055, max timestamp=1680530306063Num uniques 1122; 2023-04-03 13:58:26,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680530306055, max timestamp=1680530306063Num uniques 1009; 2023-04-03 13:58:26,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680530306055, max timestamp=1680530306063Num uniques 1122; 2023-04-03 13:58:26,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7872, suffix heap size=132904,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=172 2023-04-03 13:58:26,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-03 13:58:26,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=7872, suffix heap size=144656,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=172 2023-04-03 13:58:26,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680530306111, max timestamp=1680530306120Num uniques 1031; 2023-04-03 13:58:26,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680530306111, max timestamp=1680530306120Num uniques 1031; 2023-04-03 13:58:26,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-03 13:58:26,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-03 13:58:26,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680530306111, max timestamp=1680530306120Num uniques 1031; 2023-04-03 13:58:26,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-03 13:58:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680530306130, max timestamp=1680530306142Num uniques 1040; 2023-04-03 13:58:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680530306130, max timestamp=1680530306142Num uniques 1040; 2023-04-03 13:58:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=17860, suffix heap size=136128,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=390 2023-04-03 13:58:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=17860, suffix heap size=136128,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=390 2023-04-03 13:58:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1680530306130, max timestamp=1680530306142Num uniques 1040; 2023-04-03 13:58:26,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=17860, suffix heap size=136128,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=390 2023-04-03 13:58:26,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530366161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530366163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680530306162, max timestamp=1680530306171Num uniques 1249; 2023-04-03 13:58:26,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=6870, suffix heap size=157864,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=150 2023-04-03 13:58:26,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680530306158, max timestamp=1680530306171Num uniques 1249; 2023-04-03 13:58:26,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-04-03 13:58:26,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1680530306158, max timestamp=1680530306171Num uniques 1249; 2023-04-03 13:58:26,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-04-03 13:58:26,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680530306199, max timestamp=1680530306227Num uniques 1158; 2023-04-03 13:58:26,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680530306199, max timestamp=1680530306227Num uniques 1158; 2023-04-03 13:58:26,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-03 13:58:26,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-03 13:58:26,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680530306199, max timestamp=1680530306227Num uniques 1009; 2023-04-03 13:58:26,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=11810, suffix heap size=132904,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=258 2023-04-03 13:58:26,237 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/09184d412adf46589e539756170752e4 2023-04-03 13:58:26,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680530306271, max timestamp=1680530306287Num uniques 1117; 2023-04-03 13:58:26,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680530306271, max timestamp=1680530306287Num uniques 1117; 2023-04-03 13:58:26,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-04-03 13:58:26,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680530306271, max timestamp=1680530306287Num uniques 1117; 2023-04-03 13:58:26,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-04-03 13:58:26,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-04-03 13:58:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1680530306297, max timestamp=1680530306307Num uniques 1126; 2023-04-03 13:58:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-03 13:58:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1680530306297, max timestamp=1680530306307Num uniques 1126; 2023-04-03 13:58:26,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-03 13:58:26,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1680530306297, max timestamp=1680530306307Num uniques 1126; 2023-04-03 13:58:26,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=14926, suffix heap size=145072,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=326 2023-04-03 13:58:26,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680530306316, max timestamp=1680530306327Num uniques 1185; 2023-04-03 13:58:26,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=15340, suffix heap size=151208,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=335 2023-04-03 13:58:26,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680530306316, max timestamp=1680530306327Num uniques 1185; 2023-04-03 13:58:26,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=15340, suffix heap size=151208,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=335 2023-04-03 13:58:26,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1680530306316, max timestamp=1680530306327Num uniques 1185; 2023-04-03 13:58:26,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=15340, suffix heap size=151208,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=335 2023-04-03 13:58:26,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680530306317, max timestamp=1680530306344Num uniques 1194; 2023-04-03 13:58:26,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=13464, suffix heap size=152144,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=294 2023-04-03 13:58:26,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680530306317, max timestamp=1680530306344Num uniques 1194; 2023-04-03 13:58:26,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=13464, suffix heap size=152144,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=294 2023-04-03 13:58:26,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1680530306317, max timestamp=1680530306344Num uniques 1194; 2023-04-03 13:58:26,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=13464, suffix heap size=152144,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=294 2023-04-03 13:58:26,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1680530306352, max timestamp=1680530306362Num uniques 1153; 2023-04-03 13:58:26,364 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=13875, suffix heap size=147880,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=303 2023-04-03 13:58:26,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1680530306352, max timestamp=1680530306362Num uniques 1153; 2023-04-03 13:58:26,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=13875, suffix heap size=147880,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=303 2023-04-03 13:58:26,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=303, cellSize=13875, totalHeapSize=31752, min timestamp=1680530306352, max timestamp=1680530306362Num uniques 1153; 2023-04-03 13:58:26,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=13875, suffix heap size=147880,new segment heap size=31752 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=303 2023-04-03 13:58:26,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530366381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=362, cellSize=16572, totalHeapSize=37888, min timestamp=1680530306375, max timestamp=1680530306382Num uniques 1162; 2023-04-03 13:58:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=362, cellSize=16572, totalHeapSize=37888, min timestamp=1680530306375, max timestamp=1680530306382Num uniques 1162; 2023-04-03 13:58:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=16572, suffix heap size=148816,new segment heap size=37888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=362 2023-04-03 13:58:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=16572, suffix heap size=148816,new segment heap size=37888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=362 2023-04-03 13:58:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=362, cellSize=16572, totalHeapSize=37888, min timestamp=1680530306375, max timestamp=1680530306382Num uniques 1162; 2023-04-03 13:58:26,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=53212, new segment data size=16572, suffix heap size=148816,new segment heap size=37888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1162, new segment cells count=362 2023-04-03 13:58:26,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530366417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530366417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680530306412, max timestamp=1680530306422Num uniques 1221; 2023-04-03 13:58:26,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680530306412, max timestamp=1680530306422Num uniques 1221; 2023-04-03 13:58:26,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=55916, new segment data size=7826, suffix heap size=154952,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1221, new segment cells count=171 2023-04-03 13:58:26,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1680530306412, max timestamp=1680530306422Num uniques 1221; 2023-04-03 13:58:26,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=55916, new segment data size=7826, suffix heap size=154952,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1221, new segment cells count=171 2023-04-03 13:58:26,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=55916, new segment data size=7826, suffix heap size=154952,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1221, new segment cells count=171 2023-04-03 13:58:26,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680530306447, max timestamp=1680530306461Num uniques 1030; 2023-04-03 13:58:26,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-03 13:58:26,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680530306447, max timestamp=1680530306461Num uniques 1030; 2023-04-03 13:58:26,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1680530306447, max timestamp=1680530306461Num uniques 1030; 2023-04-03 13:58:26,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-03 13:58:26,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-04-03 13:58:26,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680530306478, max timestamp=1680530306502Num uniques 1089; 2023-04-03 13:58:26,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680530306478, max timestamp=1680530306502Num uniques 1089; 2023-04-03 13:58:26,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-03 13:58:26,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-03 13:58:26,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1680530306478, max timestamp=1680530306502Num uniques 1089; 2023-04-03 13:58:26,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-04-03 13:58:26,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=398, cellSize=18228, totalHeapSize=41632, min timestamp=1680530306519, max timestamp=1680530306550Num uniques 1148; 2023-04-03 13:58:26,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=398, cellSize=18228, totalHeapSize=41632, min timestamp=1680530306519, max timestamp=1680530306550Num uniques 1148; 2023-04-03 13:58:26,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=18228, suffix heap size=147360,new segment heap size=41632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=398 2023-04-03 13:58:26,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=18228, suffix heap size=147360,new segment heap size=41632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=398 2023-04-03 13:58:26,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=398, cellSize=18228, totalHeapSize=41632, min timestamp=1680530306519, max timestamp=1680530306550Num uniques 1148; 2023-04-03 13:58:26,552 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=18228, suffix heap size=147360,new segment heap size=41632 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=398 2023-04-03 13:58:26,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530366566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530366567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530366570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40596 deadline: 1680530366570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530366571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,647 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/4eda35df51a147b898bc667aa51e1385 2023-04-03 13:58:26,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530366672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530366672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530366673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40596 deadline: 1680530366673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530366676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530366876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530366877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530366877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530366877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:26,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:26,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530366878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,057 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/b16fcce3455a4cfebee06384d872e8c5 2023-04-03 13:58:27,061 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/09184d412adf46589e539756170752e4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/09184d412adf46589e539756170752e4 2023-04-03 13:58:27,064 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/09184d412adf46589e539756170752e4, entries=150, sequenceid=7, filesize=11.6 K 2023-04-03 13:58:27,064 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/4eda35df51a147b898bc667aa51e1385 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4eda35df51a147b898bc667aa51e1385 2023-04-03 13:58:27,067 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4eda35df51a147b898bc667aa51e1385, entries=150, sequenceid=7, filesize=11.6 K 2023-04-03 13:58:27,067 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/b16fcce3455a4cfebee06384d872e8c5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/b16fcce3455a4cfebee06384d872e8c5 2023-04-03 13:58:27,070 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/b16fcce3455a4cfebee06384d872e8c5, entries=150, sequenceid=7, filesize=11.6 K 2023-04-03 13:58:27,071 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=167.72 KB/171750 for a15553c96f61b5a8cc6d1615507e7af8 in 1261ms, sequenceid=7, compaction requested=false 2023-04-03 13:58:27,071 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:27,071 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,071 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:27,071 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:27,071 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,071 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:27,071 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:27,072 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,072 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:27,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b7aa3d2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:27,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:27,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,074 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:27,074 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:27,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,074 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:27,074 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:27,076 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,076 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,076 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,076 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:27,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:27,077 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:27,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:27,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:27,078 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:27,078 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:27,078 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,078 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,078 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,079 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,080 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,081 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,081 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:27,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:27,081 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:27,081 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,081 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,081 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,081 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:27,081 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,081 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:27,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,081 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,082 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,182 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:27,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:27,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:27,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:27,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:27,184 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:27,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:27,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:27,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,185 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,185 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:27,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:27,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:27,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:27,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:27,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,186 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:27,186 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=174.43 KB heapSize=482.81 KB 2023-04-03 13:58:27,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:27,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:58:27,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:27,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:58:27,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:27,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:58:27,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:27,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:27,188 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,188 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:27,188 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:27,188 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:27,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:27,188 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:27,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:27,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530367183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530367184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,189 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,189 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,190 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,190 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:27,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a49509[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:27,191 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:27,191 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,192 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,192 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:27,192 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:27,192 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:27,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,192 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:27,193 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:27,193 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,193 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,193 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:27,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:27,195 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:27,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:27,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:27,195 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:27,195 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:27,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530367188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,196 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530367188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530367189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,196 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,197 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,199 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,199 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,199 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,199 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:27,199 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,199 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,200 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,200 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:27,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:27,200 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:27,200 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:27,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:27,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,301 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:27,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:27,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:27,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:27,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:27,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,303 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:27,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530367297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:27,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:27,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530367298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,304 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530367298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,304 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,305 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,305 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:27,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:27,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:27,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:27,307 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,307 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:27,307 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:27,307 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:27,307 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:27,308 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,308 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:27,309 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:27,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dfddb37[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:27,309 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,310 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:27,310 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,310 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:27,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:27,310 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:27,311 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,311 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:27,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:27,313 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:27,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:27,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:27,313 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:27,313 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:27,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,313 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,314 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:27,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:27,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:27,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,318 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:27,318 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:27,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,418 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:27,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:27,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:27,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:27,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:27,421 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:27,421 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,421 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:27,421 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:27,421 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,421 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,422 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,422 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,422 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:27,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:27,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:27,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:27,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:27,423 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,423 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:27,423 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:27,423 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:27,423 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:27,424 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,424 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,424 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,424 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:27,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@228d7be[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:27,426 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:27,426 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,428 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:27,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:27,428 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:27,429 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,429 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,429 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,429 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:27,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:27,430 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:27,430 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:27,430 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:27,430 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:27,430 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:27,432 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,432 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,432 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,432 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,435 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,442 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:27,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:27,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:27,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,443 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:27,443 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:27,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530367504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530367505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530367506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,543 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:27,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:27,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:27,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:27,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:27,546 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:27,546 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,546 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:27,546 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:27,547 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,547 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,547 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,547 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,548 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,548 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:27,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:27,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:27,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:27,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:27,548 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:27,548 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,548 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:27,549 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:27,549 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:27,549 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,549 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:27,549 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:27,549 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,549 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:27,549 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:27,549 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,549 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:27,549 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:27,550 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:27,551 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:27,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c22801a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:27,551 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,552 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:27,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:27,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:27,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:27,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:27,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:27,554 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:27,554 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:27,554 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:27,554 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:27,554 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:27,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,556 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,557 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,559 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,559 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,559 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,559 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:27,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:27,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,559 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,560 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,560 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,560 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:27,560 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,560 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:27,560 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,560 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/2f0b77d287ea4f08b370a9f5a4d66a45 2023-04-03 13:58:27,607 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/a4aea3ee3f7d48d8bd40a42c3e2fa702 2023-04-03 13:58:27,660 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:27,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:27,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:27,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:27,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:27,663 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:27,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:27,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:27,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,664 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:27,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:27,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:27,665 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,665 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:27,665 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:27,665 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:27,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:27,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:27,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:27,666 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,667 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,667 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:27,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:27,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f0d5c4a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:27,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,668 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,668 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:27,668 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:27,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:27,668 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:27,669 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,669 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,669 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,669 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:27,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:27,671 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:27,671 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:27,671 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:27,671 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:27,671 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:27,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,672 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,672 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,673 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,675 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:27,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,675 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,675 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:27,675 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530367690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530367694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,775 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:27,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:27,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:27,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:27,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:27,778 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:27,778 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,778 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:27,778 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:27,778 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,778 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:27,780 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:27,780 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:27,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:27,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:27,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:27,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:27,780 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,780 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:27,781 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:27,781 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,781 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:27,781 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:27,781 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,781 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:27,781 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:27,781 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:27,783 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:27,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cba12ff[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:27,783 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,784 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,784 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:27,784 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:27,784 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:27,784 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,784 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:27,785 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,785 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:27,785 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,785 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:27,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:27,793 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:27,793 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:27,793 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:27,793 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:27,793 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:27,795 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,796 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,796 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,796 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,797 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:27,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:27,801 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:27,801 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:27,801 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:27,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530367807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530367807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,814 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:27,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530367810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,901 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:27,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:27,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:27,903 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:27,903 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,903 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:27,903 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:27,904 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,904 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,904 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,905 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,905 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:27,906 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:27,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:27,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:27,907 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,907 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:27,907 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:27,907 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:27,907 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:27,908 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,908 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,908 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,908 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:27,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186bbbf1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:27,909 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:27,909 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,910 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,910 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:27,910 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:27,910 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:27,911 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,911 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:27,912 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:27,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:27,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:27,914 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:27,914 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:27,914 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:27,914 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:27,914 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:27,915 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:27,915 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,915 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,918 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:27,919 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:27,919 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:27,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:27,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:27,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:27,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:27,920 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:27,920 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:28,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/24ac3348f482421993d0ef4114ef2456 2023-04-03 13:58:28,020 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:28,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/2f0b77d287ea4f08b370a9f5a4d66a45 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2f0b77d287ea4f08b370a9f5a4d66a45 2023-04-03 13:58:28,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:28,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:28,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:28,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:28,023 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:28,023 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:28,023 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:28,023 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:28,023 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,023 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:28,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2f0b77d287ea4f08b370a9f5a4d66a45, entries=200, sequenceid=406, filesize=14.3 K 2023-04-03 13:58:28,024 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,024 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:28,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:28,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/a4aea3ee3f7d48d8bd40a42c3e2fa702 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a4aea3ee3f7d48d8bd40a42c3e2fa702 2023-04-03 13:58:28,024 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,024 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:28,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:28,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:28,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:28,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:28,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:28,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:28,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:28,025 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:28,026 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:28,026 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:28,026 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:28,026 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:28,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a4aea3ee3f7d48d8bd40a42c3e2fa702, entries=150, sequenceid=406, filesize=11.9 K 2023-04-03 13:58:28,027 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,027 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:28,027 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:28,028 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:28,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/24ac3348f482421993d0ef4114ef2456 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/24ac3348f482421993d0ef4114ef2456 2023-04-03 13:58:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:28,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6031dab6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:28,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:28,029 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:28,030 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:28,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:28,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:28,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:28,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:28,030 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,030 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:28,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:28,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:28,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:28,031 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:28,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/24ac3348f482421993d0ef4114ef2456, entries=150, sequenceid=406, filesize=11.9 K 2023-04-03 13:58:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,033 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~482.77 KB/494352, currentSize=13.42 KB/13740 for a15553c96f61b5a8cc6d1615507e7af8 in 847ms, sequenceid=406, compaction requested=false 2023-04-03 13:58:28,033 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:28,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:28,033 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:28,033 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:28,033 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:28,033 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:28,033 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:28,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:28,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:28,035 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:28,036 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,037 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:28,037 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:28,037 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:28,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:28,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:28,037 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:28,037 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:28,038 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:28,038 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,038 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:28,038 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:28,038 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,038 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:28,038 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:28,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:28,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:28,138 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:28,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:28,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:28,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:28,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:28,140 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:28,140 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:28,141 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:28,141 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:28,141 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,141 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:28,142 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:28,142 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:28,142 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,142 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:28,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:28,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:28,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:28,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:28,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:28,143 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:28,143 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:28,143 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:28,143 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:28,143 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:28,143 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:28,143 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:28,143 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-03 13:58:28,143 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:28,143 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:28,144 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:28,144 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:28,144 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:28,144 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:28,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:28,322 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:28,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530308340, max timestamp=1680530308350Num uniques 859; 2023-04-03 13:58:28,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530308340, max timestamp=1680530308350Num uniques 859; 2023-04-03 13:58:28,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:58:28,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:58:28,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530368373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530368374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530368378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530368479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530368481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530368481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,558 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/add52675500e45018f776e9c174dd361 2023-04-03 13:58:28,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530368683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530368687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530368687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530368697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530368703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,967 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/8b1c06dc407b429a9876bce40f5004bf 2023-04-03 13:58:28,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530368990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530368991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:28,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:28,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530368991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,381 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/3e890cb90ebd4b1d8c505503e2a6321a 2023-04-03 13:58:29,385 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/add52675500e45018f776e9c174dd361 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/add52675500e45018f776e9c174dd361 2023-04-03 13:58:29,387 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/add52675500e45018f776e9c174dd361, entries=100, sequenceid=411, filesize=9.5 K 2023-04-03 13:58:29,388 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/8b1c06dc407b429a9876bce40f5004bf as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/8b1c06dc407b429a9876bce40f5004bf 2023-04-03 13:58:29,390 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/8b1c06dc407b429a9876bce40f5004bf, entries=100, sequenceid=411, filesize=9.5 K 2023-04-03 13:58:29,391 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/3e890cb90ebd4b1d8c505503e2a6321a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e890cb90ebd4b1d8c505503e2a6321a 2023-04-03 13:58:29,394 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e890cb90ebd4b1d8c505503e2a6321a, entries=100, sequenceid=411, filesize=9.5 K 2023-04-03 13:58:29,394 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=165.49 KB/169460 for a15553c96f61b5a8cc6d1615507e7af8 in 1251ms, sequenceid=411, compaction requested=true 2023-04-03 13:58:29,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:29,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:29,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:29,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:29,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:29,395 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,396 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,396 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,396 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:29,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45fb25f6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:29,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:29,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,397 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:29,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:29,397 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:29,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:29,403 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:29,403 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:29,403 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:29,403 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:29,403 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:29,409 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,409 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,409 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,410 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,411 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,421 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,421 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,421 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:29,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:29,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,422 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,422 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:29,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,422 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:29,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,422 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,423 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 13:58:29,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:29,501 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=172.20 KB heapSize=503.55 KB 2023-04-03 13:58:29,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:29,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530308364, max timestamp=1680530309500Num uniques 1018; 2023-04-03 13:58:29,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530369498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680530308364, max timestamp=1680530309500Num uniques 1018; 2023-04-03 13:58:29,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:29,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 13:58:29,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-03 13:58:29,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:29,503 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:29,503 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:29,504 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:29,504 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:58:29,522 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:29,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:29,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:29,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:29,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:29,525 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:29,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:29,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:29,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,526 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:29,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:29,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:29,527 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,527 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:29,527 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:29,527 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:29,527 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:29,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,528 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:29,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:29,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49dafa5c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:29,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,530 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,530 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:29,530 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,530 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:29,530 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:29,530 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:29,531 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,531 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:29,531 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,531 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:29,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:29,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:29,533 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:29,533 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:29,533 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:29,533 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:29,533 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:29,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,534 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,534 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:29,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40580 deadline: 1680530369531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,535 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:29,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:29,537 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:29,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,537 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,537 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,537 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:29,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530369533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,637 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:29,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:29,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:29,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530369636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:29,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:29,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:29,639 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:29,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:29,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:29,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:29,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:29,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:29,642 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,642 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:29,642 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:29,642 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:29,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:29,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:29,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:29,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cab64a5[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:29,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:29,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530369639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,645 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:29,645 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:29,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:29,645 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,646 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:29,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:29,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:29,648 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:29,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:29,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:29,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:29,648 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:29,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,650 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,650 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:29,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,652 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,652 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,652 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:29,652 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:29,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:29,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,752 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:29,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:29,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:29,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:29,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:29,755 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:29,755 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,755 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:29,755 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:29,755 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,755 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,756 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:29,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:29,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:29,757 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:29,757 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:29,757 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:29,757 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:29,758 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:29,759 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:29,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6221ebec[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:29,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:29,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,761 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:29,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:29,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:29,761 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,761 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:29,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:29,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,762 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:29,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:29,764 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:29,764 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:29,764 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:29,764 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:29,764 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:29,765 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,765 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,765 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,765 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,767 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,768 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,768 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,768 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:29,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:29,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:29,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,769 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,769 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,769 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:29,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,769 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:29,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:29,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530369840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:29,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530369847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,869 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:29,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:29,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:29,872 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:29,872 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,872 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:29,872 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:29,873 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,873 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,873 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,873 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,874 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,874 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:29,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:29,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:29,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:29,875 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,875 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:29,875 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:29,875 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:29,875 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:29,877 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,877 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,877 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,877 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:29,878 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:29,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee9a6cb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:29,878 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,878 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:29,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:29,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:29,879 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,879 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:29,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:29,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:29,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:29,881 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:29,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:29,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:29,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:29,881 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:29,882 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,882 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,882 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,884 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:29,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:29,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:29,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,885 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:29,885 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:29,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,917 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/e77bb656c2974fef8ca6ed43b0f0f724 2023-04-03 13:58:29,985 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:29,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:29,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:29,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:29,988 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:29,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:29,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:29,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,989 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,989 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:29,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:29,989 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,989 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:29,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:29,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:29,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:29,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:29,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:29,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:29,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:29,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:29,990 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,990 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:29,990 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:29,990 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:29,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:29,990 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:29,991 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,991 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:29,991 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:29,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,991 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,992 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,992 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:29,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60ffbb50[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:29,992 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:29,993 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,993 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:29,993 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:29,993 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:29,993 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,993 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:29,993 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:29,994 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,994 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:29,994 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:29,994 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:29,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:29,995 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:29,995 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:29,995 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:29,995 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:29,996 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:29,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:29,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,996 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:29,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:29,997 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:29,998 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:30,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:30,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:30,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:30,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:30,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:30,001 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:30,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,002 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:30,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:30,002 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:30,102 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:30,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:30,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:30,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:30,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:30,105 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:30,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:30,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:30,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:30,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,106 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:30,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:30,106 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,106 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,106 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:30,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:30,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:30,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:30,107 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,107 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:30,107 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:30,107 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:30,107 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:30,108 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,108 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,108 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,108 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:30,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:30,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@372675a6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:30,109 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,110 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:30,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,110 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:30,110 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:30,111 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,111 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,111 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:30,111 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:30,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:30,112 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:30,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:30,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:30,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:30,112 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:30,113 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,113 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:30,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,113 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:30,113 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,114 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,115 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:30,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,115 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:30,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:30,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:30,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:30,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:30,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:30,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,116 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:30,116 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:30,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:30,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530370149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530370155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,216 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:30,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:30,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:30,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:30,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:30,219 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:30,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:30,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:30,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:30,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,220 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:30,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:30,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,220 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:30,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:30,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:30,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:30,221 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,221 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:30,221 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:30,221 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:30,221 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:30,222 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,222 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,222 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,222 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:30,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a358625[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:30,223 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:30,223 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:30,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:30,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:30,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,225 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:30,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:30,225 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:30,226 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:30,226 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:30,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:30,228 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:30,228 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:30,228 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:30,228 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:30,228 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:30,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,229 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:30,229 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,230 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:30,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:30,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:30,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:30,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:30,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:30,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:30,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:30,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,232 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:30,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:30,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,232 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:30,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/9884099e2ad54120883a00931b27d510 2023-04-03 13:58:30,332 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:30,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:30,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:30,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:30,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:30,335 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:30,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:30,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:30,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:30,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:30,338 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:30,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:30,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:30,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:30,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:30,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:30,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:30,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:30,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:30,340 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:30,341 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,341 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:30,341 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:30,341 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:30,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:30,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:30,341 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:30,342 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,342 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,342 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,342 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:30,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69d9ac56[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:30,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=80.51 KB at sequenceid=450 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/bed4efc7009f4a44a9b2a06ea50c9a64 2023-04-03 13:58:30,350 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:30,350 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/e77bb656c2974fef8ca6ed43b0f0f724 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/e77bb656c2974fef8ca6ed43b0f0f724 2023-04-03 13:58:30,354 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:30,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:30,354 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:30,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/e77bb656c2974fef8ca6ed43b0f0f724, entries=250, sequenceid=450, filesize=16.7 K 2023-04-03 13:58:30,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:30,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/9884099e2ad54120883a00931b27d510 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/9884099e2ad54120883a00931b27d510 2023-04-03 13:58:30,359 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:30,359 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:30,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,359 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:30,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/9884099e2ad54120883a00931b27d510, entries=150, sequenceid=450, filesize=11.9 K 2023-04-03 13:58:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/bed4efc7009f4a44a9b2a06ea50c9a64 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/bed4efc7009f4a44a9b2a06ea50c9a64 2023-04-03 13:58:30,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:30,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:30,362 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:30,362 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:30,362 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:30,362 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:30,362 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:30,363 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,363 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:30,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,363 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:30,363 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,364 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/bed4efc7009f4a44a9b2a06ea50c9a64, entries=150, sequenceid=450, filesize=11.9 K 2023-04-03 13:58:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:30,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~102.87 KB/105340, heapSize ~296.61 KB/303728, currentSize=73.80 KB/75570 for a15553c96f61b5a8cc6d1615507e7af8 in 864ms, sequenceid=450, compaction requested=true 2023-04-03 13:58:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,365 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:30,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:30,365 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:30,365 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:30,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:30,365 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:30,366 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:30,366 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:30,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:30,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:30,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:30,366 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46000 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:30,366 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53320 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:30,366 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/B is initiating minor compaction (all files) 2023-04-03 13:58:30,366 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/A is initiating minor compaction (all files) 2023-04-03 13:58:30,367 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/B in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,367 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/A in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,367 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4eda35df51a147b898bc667aa51e1385, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a4aea3ee3f7d48d8bd40a42c3e2fa702, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/8b1c06dc407b429a9876bce40f5004bf, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/9884099e2ad54120883a00931b27d510] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=44.9 K 2023-04-03 13:58:30,367 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/09184d412adf46589e539756170752e4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2f0b77d287ea4f08b370a9f5a4d66a45, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/add52675500e45018f776e9c174dd361, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/e77bb656c2974fef8ca6ed43b0f0f724] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=52.1 K 2023-04-03 13:58:30,367 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4eda35df51a147b898bc667aa51e1385, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1680530305788 2023-04-03 13:58:30,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,367 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 09184d412adf46589e539756170752e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1680530305788 2023-04-03 13:58:30,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:30,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:30,367 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:30,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,367 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2f0b77d287ea4f08b370a9f5a4d66a45, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680530306563 2023-04-03 13:58:30,367 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting a4aea3ee3f7d48d8bd40a42c3e2fa702, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680530306563 2023-04-03 13:58:30,367 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:30,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,367 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:30,367 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,368 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting add52675500e45018f776e9c174dd361, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680530307187 2023-04-03 13:58:30,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:30,368 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,368 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 8b1c06dc407b429a9876bce40f5004bf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680530307187 2023-04-03 13:58:30,368 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting e77bb656c2974fef8ca6ed43b0f0f724, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680530308364 2023-04-03 13:58:30,368 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 9884099e2ad54120883a00931b27d510, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680530308373 2023-04-03 13:58:30,372 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#B#compaction#376 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:30,373 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#A#compaction#375 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:30,391 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/43517bfd6e3541a58716c466dd65b31a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/43517bfd6e3541a58716c466dd65b31a 2023-04-03 13:58:30,394 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/A of a15553c96f61b5a8cc6d1615507e7af8 into 43517bfd6e3541a58716c466dd65b31a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:30,394 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:30,395 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/A, priority=12, startTime=1680530310365; duration=0sec 2023-04-03 13:58:30,395 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:30,395 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:30,395 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46000 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:30,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/C is initiating minor compaction (all files) 2023-04-03 13:58:30,396 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/C in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,396 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/b16fcce3455a4cfebee06384d872e8c5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/24ac3348f482421993d0ef4114ef2456, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e890cb90ebd4b1d8c505503e2a6321a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/bed4efc7009f4a44a9b2a06ea50c9a64] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=44.9 K 2023-04-03 13:58:30,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting b16fcce3455a4cfebee06384d872e8c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1680530305788 2023-04-03 13:58:30,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 24ac3348f482421993d0ef4114ef2456, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680530306563 2023-04-03 13:58:30,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3e890cb90ebd4b1d8c505503e2a6321a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680530307187 2023-04-03 13:58:30,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting bed4efc7009f4a44a9b2a06ea50c9a64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680530308373 2023-04-03 13:58:30,407 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#C#compaction#377 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:30,468 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:30,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:30,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:30,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:30,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:30,470 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:30,470 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:30,470 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:30,470 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:30,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:30,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:30,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,472 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,472 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:30,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:30,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:30,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:30,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:30,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:30,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:30,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:30,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:30,472 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:30,472 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:30,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:30,473 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 13:58:30,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:30,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:30,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:30,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:30,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:30,473 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:30,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:30,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:30,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530370571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530370655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530370662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530370677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530370712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,716 DEBUG [Thread-2873] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:58:30,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530370727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:30,734 DEBUG [Thread-2877] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:58:30,782 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/ab1319f57cb24d88922a46de6b420117 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ab1319f57cb24d88922a46de6b420117 2023-04-03 13:58:30,786 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/B of a15553c96f61b5a8cc6d1615507e7af8 into ab1319f57cb24d88922a46de6b420117(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:30,786 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:30,786 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/B, priority=12, startTime=1680530310365; duration=0sec 2023-04-03 13:58:30,786 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:30,818 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/d633b3c650bd461f85f4cf82b980ec33 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/d633b3c650bd461f85f4cf82b980ec33 2023-04-03 13:58:30,822 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/C of a15553c96f61b5a8cc6d1615507e7af8 into d633b3c650bd461f85f4cf82b980ec33(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:30,822 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:30,822 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/C, priority=12, startTime=1680530310365; duration=0sec 2023-04-03 13:58:30,822 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:30,878 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/99940e22a3fa4ec1879a5018770020e6 2023-04-03 13:58:30,886 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/6f6012fca4c64fc297b7491896c7fa34 2023-04-03 13:58:30,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:30,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530370886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:31,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530371191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,320 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/6c97ce5473a741d9ac5576f2792dc60d 2023-04-03 13:58:31,323 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/99940e22a3fa4ec1879a5018770020e6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99940e22a3fa4ec1879a5018770020e6 2023-04-03 13:58:31,326 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99940e22a3fa4ec1879a5018770020e6, entries=150, sequenceid=465, filesize=11.9 K 2023-04-03 13:58:31,327 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/6f6012fca4c64fc297b7491896c7fa34 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/6f6012fca4c64fc297b7491896c7fa34 2023-04-03 13:58:31,330 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/6f6012fca4c64fc297b7491896c7fa34, entries=150, sequenceid=465, filesize=11.9 K 2023-04-03 13:58:31,331 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/6c97ce5473a741d9ac5576f2792dc60d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c97ce5473a741d9ac5576f2792dc60d 2023-04-03 13:58:31,334 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c97ce5473a741d9ac5576f2792dc60d, entries=150, sequenceid=465, filesize=11.9 K 2023-04-03 13:58:31,334 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a15553c96f61b5a8cc6d1615507e7af8 in 861ms, sequenceid=465, compaction requested=false 2023-04-03 13:58:31,335 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:31,335 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:31,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:31,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:31,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:31,335 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:31,336 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:31,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:31,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:31,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:31,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:31,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7461bf06[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:31,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:31,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:31,338 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:31,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:31,338 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:31,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:31,339 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,339 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:31,339 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:31,339 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:31,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:31,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:31,340 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:31,341 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:31,341 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:31,341 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:31,341 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:31,342 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:31,342 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,342 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:31,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:31,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:31,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:31,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:31,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:31,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:31,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:31,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:31,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:31,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:31,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:31,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:31,346 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:31,346 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:31,346 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:31,346 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,346 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:31,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:31,346 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:31,346 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,446 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:31,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:31,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:31,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:31,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:31,449 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:31,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:31,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:31,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:31,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,449 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:31,450 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:31,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:31,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,450 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:31,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:31,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:31,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:31,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:31,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:31,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:31,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:31,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:31,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:31,451 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:31,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:31,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:31,451 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 13:58:31,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:31,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:31,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:31,451 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:31,452 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:31,452 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:31,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:31,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:31,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:31,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530371700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:31,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530371700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:31,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530371701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,804 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:31,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530371802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:31,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530371807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:31,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530371807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:31,857 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/2343d435eec840e8b3aff836824d8b3e 2023-04-03 13:58:31,865 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/3deb81a607644bda9353bf479a92532b 2023-04-03 13:58:32,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530372005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530372015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40582 deadline: 1680530372016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,274 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/5cd8410dc2714fac8fb0665cde23c760 2023-04-03 13:58:32,278 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/2343d435eec840e8b3aff836824d8b3e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2343d435eec840e8b3aff836824d8b3e 2023-04-03 13:58:32,281 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2343d435eec840e8b3aff836824d8b3e, entries=150, sequenceid=485, filesize=11.9 K 2023-04-03 13:58:32,281 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/3deb81a607644bda9353bf479a92532b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3deb81a607644bda9353bf479a92532b 2023-04-03 13:58:32,285 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3deb81a607644bda9353bf479a92532b, entries=150, sequenceid=485, filesize=11.9 K 2023-04-03 13:58:32,285 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/5cd8410dc2714fac8fb0665cde23c760 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5cd8410dc2714fac8fb0665cde23c760 2023-04-03 13:58:32,288 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5cd8410dc2714fac8fb0665cde23c760, entries=150, sequenceid=485, filesize=11.9 K 2023-04-03 13:58:32,288 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a15553c96f61b5a8cc6d1615507e7af8 in 837ms, sequenceid=485, compaction requested=true 2023-04-03 13:58:32,289 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:32,289 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,289 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:32,289 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:32,289 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,290 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:32,290 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:32,291 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:32,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ee6f69[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:32,292 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:32,292 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,293 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,293 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:32,293 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:32,293 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:32,293 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,293 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:32,294 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,294 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,294 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,294 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:32,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:32,295 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:32,295 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:32,295 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:32,295 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:32,296 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:32,297 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,297 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,297 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,298 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,299 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,300 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,300 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,300 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,300 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,300 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,300 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:32,300 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:32,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:32,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,300 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,301 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,301 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,301 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:32,301 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:32,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:32,315 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 13:58:32,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:32,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:32,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:32,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:32,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:32,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:32,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530372343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530372345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530372346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,401 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:32,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:32,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:32,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:32,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:32,403 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:32,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:32,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:32,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,405 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,405 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:32,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:32,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:32,406 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,406 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:32,406 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:32,406 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:32,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:32,407 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,407 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,407 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,407 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:32,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61cdbff[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:32,408 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:32,408 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,408 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:32,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:32,409 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,409 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:32,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:32,409 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:32,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,409 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:32,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:32,411 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:32,411 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:32,411 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:32,411 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:32,411 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:32,411 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,411 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,412 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,412 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,413 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:32,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:32,415 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,415 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,417 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:32,417 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:32,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530372449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530372450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530372456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,517 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:32,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:32,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:32,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:32,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:32,519 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:32,519 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,519 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:32,519 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:32,520 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,520 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,520 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,520 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:32,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:32,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:32,522 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,522 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:32,522 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:32,522 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:32,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:32,523 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,523 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:32,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22037c76[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:32,524 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:32,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,525 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,525 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:32,525 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:32,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,525 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:32,525 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:32,526 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,526 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:32,526 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,526 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:32,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:32,529 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:32,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:32,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:32,529 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:32,529 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:32,530 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,530 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,530 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,531 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,531 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:32,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:32,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,533 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,533 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:32,533 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,533 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,633 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:32,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:32,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:32,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:32,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:32,636 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:32,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:32,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:32,636 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,637 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,638 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:32,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:32,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:32,639 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,639 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:32,639 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:32,639 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:32,639 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:32,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:32,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:32,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b637ebf[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:32,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:32,641 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,642 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:32,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:32,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:32,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:32,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:32,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:32,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:32,644 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:32,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:32,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:32,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:32,644 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:32,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,647 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,647 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,649 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:32,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:32,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,650 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,650 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,650 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,650 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:32,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,650 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:32,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530372658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530372659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530372661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,722 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/1b9b07bd589a462f8bb9522c45fd4af6 2023-04-03 13:58:32,750 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:32,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:32,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:32,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:32,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:32,752 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:32,753 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,753 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:32,753 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:32,753 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,753 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,754 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:32,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:32,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:32,755 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,755 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:32,755 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:32,755 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:32,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:32,756 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,756 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,756 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,756 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:32,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5875c79d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:32,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:32,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,758 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,758 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:32,758 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:32,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:32,758 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:32,759 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,759 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:32,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:32,761 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:32,761 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:32,761 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:32,761 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:32,761 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:32,761 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,763 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,770 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:32,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:32,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,779 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:32,779 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:32,779 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,880 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:32,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:32,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:32,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:32,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:32,882 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:32,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:32,882 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:32,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,883 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,883 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:32,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:32,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:32,885 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:32,885 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:32,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:32,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:32,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:32,885 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:32,885 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:32,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:32,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:32,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:32,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:32,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:32,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:32,886 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:32,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38563c4f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:32,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:32,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:32,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:32,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:32,889 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,889 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:32,889 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:32,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,889 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:32,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:32,891 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:32,891 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:32,891 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:32,891 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:32,891 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:32,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:32,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:32,893 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:32,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,894 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:32,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,896 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:32,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:32,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:32,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:32,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:32,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:32,896 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:32,896 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:32,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:32,897 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,897 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,897 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:32,897 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:32,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530372963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530372965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:32,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530372966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:32,997 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:32,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:32,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:32,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:32,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:32,999 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:32,999 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:32,999 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:32,999 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:33,000 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,000 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,000 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,000 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,001 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,001 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:33,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:33,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:33,002 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,002 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:33,002 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:33,002 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:33,003 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,003 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,003 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,003 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:33,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f17ff[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:33,004 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:33,004 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,005 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:33,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:33,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:33,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:33,006 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,006 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:33,006 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,006 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:33,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:33,008 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:33,008 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:33,008 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:33,008 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:33,008 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:33,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,009 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,010 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:33,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:33,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:33,012 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,013 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,013 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:33,013 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:33,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,113 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:33,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:33,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:33,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:33,116 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:33,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:33,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:33,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,117 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:33,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:33,118 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:33,118 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:33,118 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:33,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:33,119 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,119 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:33,119 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:33,119 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:33,119 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:33,120 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,120 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:33,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41d917e7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:33,121 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:33,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,122 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:33,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:33,122 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:33,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:33,123 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:33,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,123 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:33,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:33,125 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:33,125 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:33,125 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:33,125 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:33,125 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:33,126 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,126 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,126 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,126 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,127 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,129 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:33,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:33,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,129 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:33,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:33,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,129 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:33,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,129 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/e2f2462c36184d92a4072a98443bed58 2023-04-03 13:58:33,229 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:33,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:33,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:33,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:33,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:33,232 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:33,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:33,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:33,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,233 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,233 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,233 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,233 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:33,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:33,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:33,234 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,234 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:33,234 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:33,234 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:33,234 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:33,235 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,235 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,235 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:33,236 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:33,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e6696e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:33,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:33,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,237 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,237 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:33,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:33,237 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:33,237 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:33,238 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,238 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:33,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:33,240 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:33,240 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:33,240 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:33,240 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:33,240 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:33,240 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,240 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,240 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,242 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,243 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:33,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,243 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,243 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,243 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:33,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,344 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:33,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:33,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:33,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:33,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:33,346 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:33,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:33,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:33,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,348 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:33,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:33,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:33,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:33,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:33,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:33,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:33,349 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,349 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:33,349 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:33,349 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:33,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:33,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:33,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:33,350 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:33,351 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:33,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3edb8f99[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:33,351 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:33,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:33,352 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:33,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:33,353 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:33,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,353 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:33,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:33,355 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:33,355 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:33,355 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:33,355 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:33,355 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:33,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,356 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,357 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,359 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:33,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:33,359 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,359 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:33,359 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:33,460 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:33,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:33,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:33,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:33,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:33,462 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:33,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:33,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:33,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,464 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:33,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:33,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:33,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:33,465 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,465 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:33,465 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:33,465 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:33,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:33,466 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,466 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,466 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,466 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:33,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4792d752[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:33,467 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:33,467 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,468 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:33,468 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:33,468 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:33,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:33,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:33,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,469 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530373468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:33,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:33,471 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:33,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:33,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:33,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:33,471 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:33,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,471 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,472 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,473 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,474 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:33,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,474 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,474 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:33,474 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:33,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:33,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530373471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:33,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530373475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/734f09b0b04249a4b18c758675f0a6ac 2023-04-03 13:58:33,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/1b9b07bd589a462f8bb9522c45fd4af6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/1b9b07bd589a462f8bb9522c45fd4af6 2023-04-03 13:58:33,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/1b9b07bd589a462f8bb9522c45fd4af6, entries=200, sequenceid=500, filesize=14.3 K 2023-04-03 13:58:33,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/e2f2462c36184d92a4072a98443bed58 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e2f2462c36184d92a4072a98443bed58 2023-04-03 13:58:33,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e2f2462c36184d92a4072a98443bed58, entries=150, sequenceid=500, filesize=11.9 K 2023-04-03 13:58:33,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/734f09b0b04249a4b18c758675f0a6ac as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/734f09b0b04249a4b18c758675f0a6ac 2023-04-03 13:58:33,553 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/734f09b0b04249a4b18c758675f0a6ac, entries=150, sequenceid=500, filesize=11.9 K 2023-04-03 13:58:33,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a15553c96f61b5a8cc6d1615507e7af8 in 1239ms, sequenceid=500, compaction requested=true 2023-04-03 13:58:33,554 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:33,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:33,554 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:33,554 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:33,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:33,554 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:33,555 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:33,555 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/A is initiating minor compaction (all files) 2023-04-03 13:58:33,555 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:33,555 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/A in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,555 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/B is initiating minor compaction (all files) 2023-04-03 13:58:33,555 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/B in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,555 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/43517bfd6e3541a58716c466dd65b31a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99940e22a3fa4ec1879a5018770020e6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2343d435eec840e8b3aff836824d8b3e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/1b9b07bd589a462f8bb9522c45fd4af6] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=50.1 K 2023-04-03 13:58:33,555 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ab1319f57cb24d88922a46de6b420117, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/6f6012fca4c64fc297b7491896c7fa34, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3deb81a607644bda9353bf479a92532b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e2f2462c36184d92a4072a98443bed58] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=47.7 K 2023-04-03 13:58:33,555 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 43517bfd6e3541a58716c466dd65b31a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680530308373 2023-04-03 13:58:33,555 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ab1319f57cb24d88922a46de6b420117, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680530308373 2023-04-03 13:58:33,556 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 99940e22a3fa4ec1879a5018770020e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680530309523 2023-04-03 13:58:33,556 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 6f6012fca4c64fc297b7491896c7fa34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680530309523 2023-04-03 13:58:33,556 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2343d435eec840e8b3aff836824d8b3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680530310557 2023-04-03 13:58:33,556 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 3deb81a607644bda9353bf479a92532b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680530310557 2023-04-03 13:58:33,556 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 1b9b07bd589a462f8bb9522c45fd4af6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680530311690 2023-04-03 13:58:33,556 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e2f2462c36184d92a4072a98443bed58, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680530311690 2023-04-03 13:58:33,561 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#A#compaction#387 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:33,562 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#B#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:33,574 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:33,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:33,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:33,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:33,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:33,577 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:33,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:33,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:33,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,578 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,581 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/a9f85332739c479bb2d09d4d334e6e91 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a9f85332739c479bb2d09d4d334e6e91 2023-04-03 13:58:33,584 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,585 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,585 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/B of a15553c96f61b5a8cc6d1615507e7af8 into a9f85332739c479bb2d09d4d334e6e91(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:33,585 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:33,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,585 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/B, priority=12, startTime=1680530313554; duration=0sec 2023-04-03 13:58:33,585 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,585 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:33,585 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:33,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,585 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:58:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:33,586 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:58:33,586 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/C is initiating minor compaction (all files) 2023-04-03 13:58:33,586 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/C in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,586 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/d633b3c650bd461f85f4cf82b980ec33, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c97ce5473a741d9ac5576f2792dc60d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5cd8410dc2714fac8fb0665cde23c760, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/734f09b0b04249a4b18c758675f0a6ac] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=47.7 K 2023-04-03 13:58:33,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:33,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:33,587 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:33,587 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,587 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:33,587 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d633b3c650bd461f85f4cf82b980ec33, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=450, earliestPutTs=1680530308373 2023-04-03 13:58:33,587 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:33,587 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:33,587 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,587 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:33,587 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 6c97ce5473a741d9ac5576f2792dc60d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680530309523 2023-04-03 13:58:33,587 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 13:58:33,587 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:33,587 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5cd8410dc2714fac8fb0665cde23c760, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680530310557 2023-04-03 13:58:33,587 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:33,587 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:33,588 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 734f09b0b04249a4b18c758675f0a6ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680530311690 2023-04-03 13:58:33,588 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:33,588 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:33,588 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:33,599 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/0257d3b23dd34e0d9cff3be22f2f2bd2 2023-04-03 13:58:33,599 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#C#compaction#390 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:33,610 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/fda9d49b34254537976052d5d3532969 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/fda9d49b34254537976052d5d3532969 2023-04-03 13:58:33,613 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/f75a35d24d1045359a0f0428f202d55b 2023-04-03 13:58:33,614 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/C of a15553c96f61b5a8cc6d1615507e7af8 into fda9d49b34254537976052d5d3532969(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:33,614 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:33,614 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/C, priority=12, startTime=1680530313554; duration=0sec 2023-04-03 13:58:33,614 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:33,632 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/5743cb4c30bd4a8794d548ad311e7db4 2023-04-03 13:58:33,636 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/0257d3b23dd34e0d9cff3be22f2f2bd2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/0257d3b23dd34e0d9cff3be22f2f2bd2 2023-04-03 13:58:33,641 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/0257d3b23dd34e0d9cff3be22f2f2bd2, entries=150, sequenceid=518, filesize=11.9 K 2023-04-03 13:58:33,641 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/f75a35d24d1045359a0f0428f202d55b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/f75a35d24d1045359a0f0428f202d55b 2023-04-03 13:58:33,645 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/f75a35d24d1045359a0f0428f202d55b, entries=150, sequenceid=518, filesize=11.9 K 2023-04-03 13:58:33,645 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/5743cb4c30bd4a8794d548ad311e7db4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5743cb4c30bd4a8794d548ad311e7db4 2023-04-03 13:58:33,665 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5743cb4c30bd4a8794d548ad311e7db4, entries=150, sequenceid=518, filesize=11.9 K 2023-04-03 13:58:33,666 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for a15553c96f61b5a8cc6d1615507e7af8 in 79ms, sequenceid=518, compaction requested=false 2023-04-03 13:58:33,666 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:33,666 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:33,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:33,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,668 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:33,668 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:33,679 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,679 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,679 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,679 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:33,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:33,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4efd8668[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:33,681 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:33,682 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,684 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,684 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:33,684 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:33,684 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:33,684 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,685 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:33,685 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:33,685 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,685 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:33,685 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:33,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:33,687 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:33,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:33,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:33,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:33,687 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:33,688 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,688 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,688 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,690 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,691 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,691 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,691 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:33,691 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:33,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:33,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,692 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:33,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,692 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:33,792 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:33,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:33,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:33,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:33,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:33,795 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:33,795 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,795 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:33,795 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:33,795 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,795 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,796 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,796 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,796 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,796 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:33,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:33,797 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:33,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:33,797 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,797 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:33,797 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:33,797 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:33,797 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:33,798 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,798 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,799 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ad42bc0[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:33,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:33,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,800 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:33,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:33,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:33,800 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:33,801 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:33,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:33,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:33,802 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:33,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:33,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:33,803 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:33,803 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:33,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,803 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,804 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,805 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:33,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,806 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:33,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:33,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,807 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:33,808 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:33,908 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:33,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:33,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:33,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:33,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:33,910 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:33,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:33,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:33,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,910 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,911 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,911 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,911 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,911 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:33,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:33,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:33,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:33,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:33,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:33,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:33,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:33,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:33,912 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:33,912 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,912 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:33,913 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:33,913 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:33,913 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:33,913 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:33,913 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,913 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:33,913 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:33,913 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,913 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,914 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,914 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:33,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2891c244[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:33,915 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:33,915 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,916 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,916 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:33,916 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:33,916 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:33,916 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,917 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:33,917 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,917 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:33,917 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,917 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:33,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:33,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:33,920 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:33,920 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:33,920 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:33,920 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:33,920 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:33,921 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:33,921 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,921 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:33,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,922 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:33,923 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:33,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:33,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:33,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:33,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:33,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:33,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:33,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:33,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:33,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:33,924 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:33,925 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:33,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:33,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:33,925 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:33,925 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:33,971 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/179688ebc78d469cae381fd71fe7cff5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/179688ebc78d469cae381fd71fe7cff5 2023-04-03 13:58:33,974 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/A of a15553c96f61b5a8cc6d1615507e7af8 into 179688ebc78d469cae381fd71fe7cff5(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:33,974 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:33,974 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/A, priority=12, startTime=1680530313554; duration=0sec 2023-04-03 13:58:33,974 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:34,025 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,027 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,027 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,027 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,027 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,028 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,028 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,028 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,028 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,028 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,028 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,029 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,029 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,029 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,029 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,029 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,029 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,030 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,030 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:34,030 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,030 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,031 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,031 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,031 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,031 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e8b843[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,033 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,033 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,034 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,034 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,034 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,034 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,035 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,036 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,037 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,038 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,038 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,038 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,039 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:34,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,042 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,042 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,042 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,042 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,042 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,042 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,043 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,043 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:34,143 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,146 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,146 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,146 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,146 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,146 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,146 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,147 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,147 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,147 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,147 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,148 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,148 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,148 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,148 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,148 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,148 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,149 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,149 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:34,149 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,149 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,149 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,149 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,149 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,149 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,150 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,150 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,150 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,150 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b04afd0[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,153 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,154 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,154 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,154 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,155 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,159 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,159 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,159 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,159 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,159 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,169 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,170 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,171 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,171 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,172 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:34,171 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,172 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,172 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,272 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,275 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,275 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,275 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,275 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,275 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,275 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,276 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,276 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,276 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,276 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,277 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,277 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,277 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:34,277 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,278 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,278 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,279 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,279 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3065155b[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,280 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,280 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,281 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,281 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,282 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,282 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,284 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,284 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,284 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,284 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,284 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,285 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,285 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,286 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,288 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,288 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,288 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,288 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,388 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,391 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,392 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,392 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,392 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,393 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:34,394 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,394 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,395 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,395 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c072f1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,398 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,398 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,399 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,399 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,399 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,399 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,400 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,400 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,400 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,400 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,401 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,401 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,401 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,401 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,402 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,404 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,405 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,405 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:34,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:34,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:58:34,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:34,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:34,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:34,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:34,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:34,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:34,505 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,507 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,508 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,509 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,509 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,509 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:34,509 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,510 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,510 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,511 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a56faca[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,511 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,512 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,512 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,512 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,513 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,513 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,513 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,513 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,515 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,515 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,516 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,516 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,516 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,519 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,519 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,519 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,519 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:34,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680530314504, max timestamp=1680530314521Num uniques 859; 2023-04-03 13:58:34,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 13:58:34,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680530314504, max timestamp=1680530314521Num uniques 859; 2023-04-03 13:58:34,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680530314504, max timestamp=1680530314521Num uniques 859; 2023-04-03 13:58:34,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 13:58:34,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 13:58:34,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530374547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530374548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530374549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,619 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,622 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,623 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,624 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,624 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,624 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:34,624 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,625 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,625 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d2ba7f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,627 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,627 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,628 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,628 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,630 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,630 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,630 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,630 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,630 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,631 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,631 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,631 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:34,634 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,634 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,634 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,634 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530374650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530374654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530374656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,734 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,737 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,737 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,738 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:34,740 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,741 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,741 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c05b273[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,742 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,743 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,743 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,744 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,745 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,745 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,746 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,746 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,746 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,746 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,747 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,749 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,749 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,749 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:34,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530374751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,758 DEBUG [Thread-2873] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:58:34,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530374764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,770 DEBUG [Thread-2877] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8201 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:58:34,849 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,852 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,853 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,853 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,853 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,853 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,855 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,855 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,855 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:34,855 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,856 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,856 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@594716f8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,857 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,857 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,857 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,858 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,858 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,858 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,859 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,859 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530374855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,861 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,861 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530374858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,862 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,862 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,862 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,864 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:34,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:34,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530374861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,866 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,866 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:34,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/59fe5a7924d148d1a85c82fc78ba170f 2023-04-03 13:58:34,967 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:34,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:34,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:34,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:34,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:34,969 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:34,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:34,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:34,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,970 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,970 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:34,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:34,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:34,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:34,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:34,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:34,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:34,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:34,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:34,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:34,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44da37b0[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:34,973 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,974 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:34,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:34,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:34,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:34,975 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:34,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:34,976 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:34,976 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:34,976 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:34,976 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:34,976 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:34,977 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,977 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,977 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:34,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,980 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:34,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:34,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:34,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:34,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:34,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:34,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:34,981 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:34,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:34,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:34,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:34,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:34,981 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:34,981 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:35,081 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:35,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:35,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:35,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:35,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:35,084 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:35,084 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,084 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:35,084 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:35,084 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,084 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,085 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:35,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:35,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:35,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:35,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:35,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:35,086 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,086 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:35,087 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:35,087 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:35,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,087 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:35,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51a7c16[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:35,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:35,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,089 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,089 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:35,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:35,089 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:35,089 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:35,090 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:35,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:35,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:35,092 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:35,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:35,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:35,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:35,092 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:35,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,097 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:35,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:35,097 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:35,098 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:35,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:35,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530375161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:35,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530375162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:35,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530375167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,198 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:35,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:35,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:35,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:35,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:35,200 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:35,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:35,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:35,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,202 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,202 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,202 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,202 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:35,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:35,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:35,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:35,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:35,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:35,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:35,203 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:35,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:35,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,204 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:35,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:35,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64edd0ec[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:35,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:35,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,206 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:35,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:35,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:35,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:35,207 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,207 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:35,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:35,209 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:35,209 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:35,209 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:35,209 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:35,209 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:35,209 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,210 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,211 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:35,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:35,213 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:35,213 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:35,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:35,212 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,212 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/4380124443db4b7dbdc2523af6dfaf4a 2023-04-03 13:58:35,313 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:35,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:35,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:35,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:35,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:35,315 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:35,315 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:35,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:35,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,316 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:35,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:35,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:35,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:35,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:35,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:35,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:35,318 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,318 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:35,318 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:35,318 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:35,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:35,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:35,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:35,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:35,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:35,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57cea5f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:35,321 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:35,321 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:35,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:35,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:35,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,323 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,323 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,323 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:35,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:35,325 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:35,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:35,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:35,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:35,325 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:35,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,326 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,327 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,330 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:35,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:35,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:35,330 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:35,330 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:35,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,430 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:35,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:35,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:35,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:35,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:35,433 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:35,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:35,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:35,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,435 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,435 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,435 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,435 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:35,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:35,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:35,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:35,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:35,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:35,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:35,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:35,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:35,436 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,436 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:35,436 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:35,436 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:35,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:35,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,437 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:35,437 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:35,439 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,439 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,440 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,440 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:35,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:35,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@108e2af7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:35,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:35,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,441 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:35,441 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:35,441 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:35,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:35,442 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:35,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:35,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:35,443 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:35,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:35,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:35,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:35,444 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:35,444 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,444 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,444 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,445 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:35,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:35,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,447 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:35,447 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:35,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,547 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:35,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:35,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:35,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:35,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:35,550 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:35,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:35,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:35,551 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,551 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,551 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,551 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:35,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:35,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:35,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:35,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:35,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:35,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:35,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:35,552 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:35,552 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,553 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:35,553 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:35,553 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,553 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:35,553 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:35,553 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,553 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:35,553 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:35,553 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,554 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:35,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46221e3c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:35,555 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:35,555 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,555 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:35,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:35,555 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,556 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:35,555 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:35,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,556 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:35,556 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,556 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:35,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:35,558 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:35,558 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:35,558 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:35,558 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:35,558 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:35,559 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,559 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,559 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,559 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,560 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:35,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:35,561 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,561 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:35,561 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,561 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,661 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:35,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:35,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:35,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:35,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:35,663 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:35,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:35,663 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:35,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,664 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,665 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,665 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:35,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:35,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:35,666 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,666 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:35,666 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:35,666 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:35,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:35,667 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,667 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,667 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,667 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:35,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530375665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:35,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cc2cca4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:35,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:35,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,669 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:35,669 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:35,669 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,669 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:35,670 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,670 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:35,670 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,670 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:35,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:35,672 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:35,672 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:35,672 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:35,672 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:35,672 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:35,673 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,673 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:35,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,673 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:35,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530375667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:35,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,673 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,674 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:35,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:35,676 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:35,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:35,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530375673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,676 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,676 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,676 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,676 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,676 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,676 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:35,676 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,676 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:35,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:35,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,677 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:35,677 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:35,677 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:35,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:35,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/23aa35df157f478d970cc80c89e2031e 2023-04-03 13:58:35,714 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/59fe5a7924d148d1a85c82fc78ba170f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/59fe5a7924d148d1a85c82fc78ba170f 2023-04-03 13:58:35,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/59fe5a7924d148d1a85c82fc78ba170f, entries=250, sequenceid=537, filesize=16.7 K 2023-04-03 13:58:35,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/4380124443db4b7dbdc2523af6dfaf4a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4380124443db4b7dbdc2523af6dfaf4a 2023-04-03 13:58:35,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4380124443db4b7dbdc2523af6dfaf4a, entries=150, sequenceid=537, filesize=11.9 K 2023-04-03 13:58:35,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/23aa35df157f478d970cc80c89e2031e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/23aa35df157f478d970cc80c89e2031e 2023-04-03 13:58:35,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/23aa35df157f478d970cc80c89e2031e, entries=150, sequenceid=537, filesize=11.9 K 2023-04-03 13:58:35,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a15553c96f61b5a8cc6d1615507e7af8 in 1242ms, sequenceid=537, compaction requested=true 2023-04-03 13:58:35,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:35,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:35,729 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:58:35,729 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:58:35,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:35,729 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:35,730 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41712 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:58:35,730 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/A is initiating minor compaction (all files) 2023-04-03 13:58:35,730 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36832 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:58:35,730 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/A in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,730 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/B is initiating minor compaction (all files) 2023-04-03 13:58:35,730 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/179688ebc78d469cae381fd71fe7cff5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/0257d3b23dd34e0d9cff3be22f2f2bd2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/59fe5a7924d148d1a85c82fc78ba170f] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=40.7 K 2023-04-03 13:58:35,730 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/B in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,730 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a9f85332739c479bb2d09d4d334e6e91, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/f75a35d24d1045359a0f0428f202d55b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4380124443db4b7dbdc2523af6dfaf4a] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=36.0 K 2023-04-03 13:58:35,730 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 179688ebc78d469cae381fd71fe7cff5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680530311690 2023-04-03 13:58:35,730 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting a9f85332739c479bb2d09d4d334e6e91, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680530311690 2023-04-03 13:58:35,731 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 0257d3b23dd34e0d9cff3be22f2f2bd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680530312342 2023-04-03 13:58:35,731 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f75a35d24d1045359a0f0428f202d55b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680530312342 2023-04-03 13:58:35,731 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 59fe5a7924d148d1a85c82fc78ba170f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530314476 2023-04-03 13:58:35,731 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4380124443db4b7dbdc2523af6dfaf4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530314481 2023-04-03 13:58:35,736 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#A#compaction#396 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:35,737 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#B#compaction#397 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:35,749 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/268321b0193440609f14178af1f312e7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/268321b0193440609f14178af1f312e7 2023-04-03 13:58:35,753 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/B of a15553c96f61b5a8cc6d1615507e7af8 into 268321b0193440609f14178af1f312e7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:35,753 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:35,753 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/B, priority=13, startTime=1680530315729; duration=0sec 2023-04-03 13:58:35,753 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:35,753 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:58:35,754 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36832 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:58:35,754 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/C is initiating minor compaction (all files) 2023-04-03 13:58:35,754 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/C in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,754 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/fda9d49b34254537976052d5d3532969, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5743cb4c30bd4a8794d548ad311e7db4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/23aa35df157f478d970cc80c89e2031e] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=36.0 K 2023-04-03 13:58:35,754 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fda9d49b34254537976052d5d3532969, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1680530311690 2023-04-03 13:58:35,755 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 5743cb4c30bd4a8794d548ad311e7db4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680530312342 2023-04-03 13:58:35,755 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 23aa35df157f478d970cc80c89e2031e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530314481 2023-04-03 13:58:35,760 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#C#compaction#398 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:35,777 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:35,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:35,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:35,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:35,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:35,779 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:35,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,780 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:35,780 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:35,780 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,780 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:35,780 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,781 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:35,781 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:35,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:35,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:35,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:35,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:35,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:35,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:35,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:35,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:35,782 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:35,782 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:35,782 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:35,782 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-03 13:58:35,782 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:35,782 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:35,782 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:35,782 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:35,782 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:35,782 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:36,149 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/7a39c390590a44aa8911f126c91fd5c4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/7a39c390590a44aa8911f126c91fd5c4 2023-04-03 13:58:36,152 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/A of a15553c96f61b5a8cc6d1615507e7af8 into 7a39c390590a44aa8911f126c91fd5c4(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:36,152 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:36,152 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/A, priority=13, startTime=1680530315729; duration=0sec 2023-04-03 13:58:36,152 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:36,169 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/941baa34f5704c2a9b337d6f9261da15 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/941baa34f5704c2a9b337d6f9261da15 2023-04-03 13:58:36,172 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/C of a15553c96f61b5a8cc6d1615507e7af8 into 941baa34f5704c2a9b337d6f9261da15(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:36,172 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:36,172 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/C, priority=13, startTime=1680530315729; duration=0sec 2023-04-03 13:58:36,172 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:36,188 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/063a5a8d8e1b47c69dda0b0b8371f8a0 2023-04-03 13:58:36,197 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/2a08dfbceabf47b890e6a45212e1dde9 2023-04-03 13:58:36,606 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/5b08160a8a4b4d25932afda5ac261917 2023-04-03 13:58:36,609 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/063a5a8d8e1b47c69dda0b0b8371f8a0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/063a5a8d8e1b47c69dda0b0b8371f8a0 2023-04-03 13:58:36,612 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/063a5a8d8e1b47c69dda0b0b8371f8a0, entries=150, sequenceid=572, filesize=11.9 K 2023-04-03 13:58:36,612 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/2a08dfbceabf47b890e6a45212e1dde9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/2a08dfbceabf47b890e6a45212e1dde9 2023-04-03 13:58:36,614 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/2a08dfbceabf47b890e6a45212e1dde9, entries=150, sequenceid=572, filesize=11.9 K 2023-04-03 13:58:36,615 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/5b08160a8a4b4d25932afda5ac261917 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5b08160a8a4b4d25932afda5ac261917 2023-04-03 13:58:36,618 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5b08160a8a4b4d25932afda5ac261917, entries=150, sequenceid=572, filesize=11.9 K 2023-04-03 13:58:36,618 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for a15553c96f61b5a8cc6d1615507e7af8 in 836ms, sequenceid=572, compaction requested=false 2023-04-03 13:58:36,618 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:36,618 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:36,618 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:36,619 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:36,621 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,621 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:36,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e8ee971[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:36,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:36,622 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,623 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:36,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:36,623 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:36,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,623 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:36,624 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:36,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,624 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:36,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:36,626 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:36,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:36,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:36,626 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:36,626 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:36,627 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,627 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,627 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,627 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:36,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,628 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,629 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,630 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:36,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:36,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:36,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:36,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:36,630 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:36,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,630 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:36,630 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:36,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:36,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:36,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:36,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:36,686 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:58:36,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:36,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:36,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:36,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:36,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:36,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:36,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/b237e0171d88476c928aacdba4777c73 2023-04-03 13:58:36,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530316699, max timestamp=1680530316719Num uniques 859; 2023-04-03 13:58:36,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530316699, max timestamp=1680530316719Num uniques 859; 2023-04-03 13:58:36,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680530316699, max timestamp=1680530316719Num uniques 859; 2023-04-03 13:58:36,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:58:36,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:58:36,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 13:58:36,730 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:36,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:36,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:36,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:36,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:36,733 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:36,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:36,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:36,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:36,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,733 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,734 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:36,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:36,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,734 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:36,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:36,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:36,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:36,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:36,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:36,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:36,735 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:36,735 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:36,736 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,736 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,737 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:36,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f36b7c5[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:36,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:36,738 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,739 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,739 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:36,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:36,739 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:36,740 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:36,740 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:36,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:36,741 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:36,741 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:36,741 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:36,741 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:36,741 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:36,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,742 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,743 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:36,745 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:36,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:36,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:36,745 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:36,745 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:36,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,745 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:36,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530376749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530376751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530376750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,845 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:36,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:36,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:36,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:36,848 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:36,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:36,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:36,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:36,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,848 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,849 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:36,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:36,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,849 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:36,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:36,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:36,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:36,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:36,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:36,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:36,850 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:36,850 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:36,851 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,851 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,851 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,851 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:36,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b383e1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:36,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:36,852 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,854 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:36,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:36,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,854 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:36,854 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:36,855 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:36,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,855 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:36,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:36,856 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:36,856 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:36,857 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:36,857 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:36,857 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:36,857 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,857 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,857 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:36,858 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:36,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530376857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,860 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:36,863 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:36,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:36,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:36,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:36,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:36,864 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:36,864 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:36,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:36,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:36,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:36,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:36,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530376866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,871 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:36,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530376866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,964 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:36,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:36,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:36,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:36,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:36,967 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:36,967 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:36,967 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:36,967 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:36,968 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,968 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,968 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,968 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:36,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:36,968 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,968 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:36,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:36,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:36,969 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:36,969 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:36,969 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:36,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:36,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:36,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:36,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:36,971 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:36,972 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:36,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:36,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:36,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:36,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:36,972 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,972 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,973 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,973 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:36,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43cedb9f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:36,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:36,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,974 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,974 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:36,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:36,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:36,975 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,975 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:36,975 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:36,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:36,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:36,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:36,977 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:36,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:36,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:36,977 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:36,977 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:36,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:36,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:36,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:36,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:36,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:36,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:36,981 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:36,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:36,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:36,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:36,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:36,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:36,982 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:36,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:36,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:36,982 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:36,982 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530377062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530377071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530377072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,082 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:37,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:37,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:37,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:37,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:37,085 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:37,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:37,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:37,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,085 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,086 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:37,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:37,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:37,087 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,087 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:37,087 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:37,087 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:37,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:37,088 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,088 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:37,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad277f6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:37,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:37,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,090 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:37,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:37,090 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:37,090 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,090 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:37,090 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:37,091 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:37,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:37,093 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:37,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:37,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:37,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:37,093 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:37,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,094 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,094 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:37,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,096 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:37,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:37,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:37,097 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:37,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:37,097 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:37,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,097 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:37,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/33b0e0ccfaca47f3bfa20a08b7d8d8be 2023-04-03 13:58:37,197 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:37,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:37,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:37,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:37,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:37,200 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:37,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:37,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:37,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,200 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,201 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:37,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:37,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:37,202 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,202 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:37,202 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:37,202 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:37,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:37,203 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,204 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:37,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@602073f6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:37,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:37,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,205 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,205 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:37,205 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:37,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,205 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:37,205 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:37,206 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:37,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:37,206 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:37,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:37,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:37,208 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:37,208 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:37,208 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:37,208 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:37,208 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:37,209 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,209 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,209 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:37,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,212 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,213 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:37,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:37,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:37,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:37,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:37,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,214 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,214 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:37,214 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:37,214 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,214 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:37,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:37,214 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,314 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:37,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:37,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:37,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:37,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:37,316 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:37,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:37,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:37,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:37,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:37,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:37,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:37,319 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,319 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:37,319 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:37,319 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:37,319 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:37,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:37,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:37,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65894898[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:37,321 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:37,321 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:37,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:37,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:37,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:37,323 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,323 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:37,323 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:37,323 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:37,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:37,325 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:37,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:37,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:37,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:37,325 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:37,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:37,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:37,327 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:37,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:37,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:37,330 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:37,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:37,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:37,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:37,330 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:37,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,330 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,330 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:37,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530377365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530377377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530377378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,430 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:37,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:37,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:37,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:37,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:37,433 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:37,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:37,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:37,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,434 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:37,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:37,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:37,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:37,435 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,435 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:37,435 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:37,435 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:37,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:37,436 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,436 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,437 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:37,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:37,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50ae6595[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:37,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:37,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:37,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:37,438 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:37,439 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,439 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,439 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:37,439 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:37,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:37,441 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:37,442 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:37,442 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:37,442 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:37,442 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:37,442 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:37,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:37,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,443 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:37,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:37,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:37,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,445 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,445 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:37,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:37,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:37,445 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:37,445 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:37,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:37,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:37,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:37,512 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/ada5ac3b58794069867d6f38ea741184 2023-04-03 13:58:37,515 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/b237e0171d88476c928aacdba4777c73 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/b237e0171d88476c928aacdba4777c73 2023-04-03 13:58:37,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/b237e0171d88476c928aacdba4777c73, entries=300, sequenceid=586, filesize=19.0 K 2023-04-03 13:58:37,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/33b0e0ccfaca47f3bfa20a08b7d8d8be as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/33b0e0ccfaca47f3bfa20a08b7d8d8be 2023-04-03 13:58:37,520 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/33b0e0ccfaca47f3bfa20a08b7d8d8be, entries=150, sequenceid=586, filesize=11.9 K 2023-04-03 13:58:37,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/ada5ac3b58794069867d6f38ea741184 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/ada5ac3b58794069867d6f38ea741184 2023-04-03 13:58:37,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/ada5ac3b58794069867d6f38ea741184, entries=150, sequenceid=586, filesize=11.9 K 2023-04-03 13:58:37,524 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for a15553c96f61b5a8cc6d1615507e7af8 in 838ms, sequenceid=586, compaction requested=true 2023-04-03 13:58:37,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:37,524 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:37,524 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:58:37,524 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:58:37,524 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:37,524 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:37,525 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44254 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:58:37,525 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/A is initiating minor compaction (all files) 2023-04-03 13:58:37,525 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:58:37,525 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/A in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,525 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/B is initiating minor compaction (all files) 2023-04-03 13:58:37,525 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/7a39c390590a44aa8911f126c91fd5c4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/063a5a8d8e1b47c69dda0b0b8371f8a0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/b237e0171d88476c928aacdba4777c73] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=43.2 K 2023-04-03 13:58:37,525 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/B in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,525 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/268321b0193440609f14178af1f312e7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/2a08dfbceabf47b890e6a45212e1dde9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/33b0e0ccfaca47f3bfa20a08b7d8d8be] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=36.1 K 2023-04-03 13:58:37,525 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 7a39c390590a44aa8911f126c91fd5c4, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530314481 2023-04-03 13:58:37,526 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 268321b0193440609f14178af1f312e7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530314481 2023-04-03 13:58:37,526 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 063a5a8d8e1b47c69dda0b0b8371f8a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680530314533 2023-04-03 13:58:37,526 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2a08dfbceabf47b890e6a45212e1dde9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680530314533 2023-04-03 13:58:37,526 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting b237e0171d88476c928aacdba4777c73, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680530316679 2023-04-03 13:58:37,526 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 33b0e0ccfaca47f3bfa20a08b7d8d8be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680530316681 2023-04-03 13:58:37,530 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:37,530 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#B#compaction#406 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:37,536 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/9f3d474ba77a43819f8cd279e7f42cf2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/9f3d474ba77a43819f8cd279e7f42cf2 2023-04-03 13:58:37,539 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/A of a15553c96f61b5a8cc6d1615507e7af8 into 9f3d474ba77a43819f8cd279e7f42cf2(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:37,539 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:37,539 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/A, priority=13, startTime=1680530317524; duration=0sec 2023-04-03 13:58:37,539 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:37,540 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 13:58:37,540 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 13:58:37,540 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/C is initiating minor compaction (all files) 2023-04-03 13:58:37,540 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/C in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,540 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/941baa34f5704c2a9b337d6f9261da15, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5b08160a8a4b4d25932afda5ac261917, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/ada5ac3b58794069867d6f38ea741184] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=36.1 K 2023-04-03 13:58:37,541 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 941baa34f5704c2a9b337d6f9261da15, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680530314481 2023-04-03 13:58:37,541 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5b08160a8a4b4d25932afda5ac261917, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680530314533 2023-04-03 13:58:37,541 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting ada5ac3b58794069867d6f38ea741184, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680530316681 2023-04-03 13:58:37,545 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#C#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:37,545 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:37,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:37,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:37,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:37,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:37,548 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:37,548 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,548 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:37,548 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:37,548 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:37,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:37,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:37,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:37,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:37,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:37,551 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:37,551 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:37,551 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:37,551 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/1a981f724f8d4a69a4475b20c2d4658a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1a981f724f8d4a69a4475b20c2d4658a 2023-04-03 13:58:37,551 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:37,551 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:37,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:37,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:37,551 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-04-03 13:58:37,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:37,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:37,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:37,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:37,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:37,551 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:37,554 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/C of a15553c96f61b5a8cc6d1615507e7af8 into 1a981f724f8d4a69a4475b20c2d4658a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:37,554 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:37,554 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/C, priority=13, startTime=1680530317524; duration=0sec 2023-04-03 13:58:37,554 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:37,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:37,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:37,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530377890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530377891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530377891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,944 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/c4731d40a99349d1aefbd91037d2fe40 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c4731d40a99349d1aefbd91037d2fe40 2023-04-03 13:58:37,948 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/B of a15553c96f61b5a8cc6d1615507e7af8 into c4731d40a99349d1aefbd91037d2fe40(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:37,948 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:37,948 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/B, priority=13, startTime=1680530317524; duration=0sec 2023-04-03 13:58:37,948 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:37,958 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/74504975269841ebb8faa51d1fd58e38 2023-04-03 13:58:37,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530377995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:37,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:37,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530377995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530377999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530378200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530378201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530378203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,366 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/19e7ac9aca54420592cdf915b4d6e746 2023-04-03 13:58:38,374 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=620 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/1e340b0542b5433982c73da1aefe5cac 2023-04-03 13:58:38,377 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/74504975269841ebb8faa51d1fd58e38 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/74504975269841ebb8faa51d1fd58e38 2023-04-03 13:58:38,380 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/74504975269841ebb8faa51d1fd58e38, entries=150, sequenceid=620, filesize=11.9 K 2023-04-03 13:58:38,380 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/19e7ac9aca54420592cdf915b4d6e746 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/19e7ac9aca54420592cdf915b4d6e746 2023-04-03 13:58:38,383 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/19e7ac9aca54420592cdf915b4d6e746, entries=150, sequenceid=620, filesize=11.9 K 2023-04-03 13:58:38,384 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/1e340b0542b5433982c73da1aefe5cac as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1e340b0542b5433982c73da1aefe5cac 2023-04-03 13:58:38,387 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1e340b0542b5433982c73da1aefe5cac, entries=150, sequenceid=620, filesize=11.9 K 2023-04-03 13:58:38,387 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=60.38 KB/61830 for a15553c96f61b5a8cc6d1615507e7af8 in 836ms, sequenceid=620, compaction requested=false 2023-04-03 13:58:38,387 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:38,387 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:38,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:38,388 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:38,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:38,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:38,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:38,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:38,390 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:38,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59cad5b1[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:38,390 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:38,390 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:38,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:38,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:38,391 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:38,391 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:38,391 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:38,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:38,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:38,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:38,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:38,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:38,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:38,393 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:38,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:38,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:38,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:38,394 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:38,395 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,395 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:38,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,395 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:38,395 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:38,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,396 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:38,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:38,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:38,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:38,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:38,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:38,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:38,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:38,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:38,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:38,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:38,398 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:38,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:38,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:38,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,398 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:38,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:38,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:38,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,498 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:38,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:38,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:38,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:38,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:38,501 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:38,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:38,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:38,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:38,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:38,502 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:38,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:38,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:38,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:38,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:38,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:38,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:38,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:38,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:38,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:38,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:38,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:38,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:38,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:38,503 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:38,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:38,503 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:38,503 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:58:38,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:38,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:38,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:38,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:38,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:38,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:38,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:38,506 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:38,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530378533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530378534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530378536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530378637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530378640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530378640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530378841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530378844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:38,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530378844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:38,910 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/a70bf8abf1924b34969d47566c5e66b2 2023-04-03 13:58:38,920 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/bb53b3333e75400ca62cf810186c75c5 2023-04-03 13:58:39,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530379144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530379148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530379149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,330 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=634 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/3e5357c4370d424fbf688ca7ab9bb042 2023-04-03 13:58:39,333 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/a70bf8abf1924b34969d47566c5e66b2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a70bf8abf1924b34969d47566c5e66b2 2023-04-03 13:58:39,336 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a70bf8abf1924b34969d47566c5e66b2, entries=150, sequenceid=634, filesize=11.9 K 2023-04-03 13:58:39,336 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/bb53b3333e75400ca62cf810186c75c5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/bb53b3333e75400ca62cf810186c75c5 2023-04-03 13:58:39,340 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/bb53b3333e75400ca62cf810186c75c5, entries=150, sequenceid=634, filesize=11.9 K 2023-04-03 13:58:39,340 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/3e5357c4370d424fbf688ca7ab9bb042 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e5357c4370d424fbf688ca7ab9bb042 2023-04-03 13:58:39,343 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e5357c4370d424fbf688ca7ab9bb042, entries=150, sequenceid=634, filesize=11.9 K 2023-04-03 13:58:39,344 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a15553c96f61b5a8cc6d1615507e7af8 in 841ms, sequenceid=634, compaction requested=true 2023-04-03 13:58:39,344 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:39,344 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:39,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:39,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:39,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:39,344 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:39,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:39,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:39,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:39,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:39,345 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:39,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:39,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:39,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:39,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:39,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:39,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a96b2bd[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:39,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:39,346 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:39,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:39,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:39,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:39,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:39,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,347 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:39,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:39,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:39,348 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:39,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:39,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:39,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:39,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:39,349 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:39,349 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:39,350 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:39,350 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:39,350 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:39,350 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,350 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,350 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:39,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:39,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:39,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:39,351 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:39,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:39,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:39,361 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:39,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,377 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,377 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:39,377 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:39,377 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:39,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:39,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:39,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:39,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:39,377 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:39,377 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:39,377 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:39,377 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:39,377 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,377 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:39,378 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:39,378 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:39,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:39,477 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:39,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:39,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:39,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:39,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:39,480 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:39,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:39,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:39,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:39,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,480 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:39,481 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:39,481 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:39,481 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,481 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:39,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:39,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:39,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:39,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:39,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:39,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:39,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:39,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:39,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:39,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:39,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:39,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:39,483 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 13:58:39,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:39,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:39,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:39,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:39,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:39,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:39,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:39,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:39,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530379663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530379663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530379666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530379770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530379770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530379770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,890 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/af73738c38c74aff90624facd9f6ef00 2023-04-03 13:58:39,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530379974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530379974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:39,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:39,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530379974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530380280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530380280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530380282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,300 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/c727c4c425944b08a65f49b8a1dcfbf9 2023-04-03 13:58:40,311 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/1f88ad8073194735bcc64e3077e148d6 2023-04-03 13:58:40,316 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/af73738c38c74aff90624facd9f6ef00 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/af73738c38c74aff90624facd9f6ef00 2023-04-03 13:58:40,319 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/af73738c38c74aff90624facd9f6ef00, entries=150, sequenceid=654, filesize=11.9 K 2023-04-03 13:58:40,320 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/c727c4c425944b08a65f49b8a1dcfbf9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c727c4c425944b08a65f49b8a1dcfbf9 2023-04-03 13:58:40,322 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c727c4c425944b08a65f49b8a1dcfbf9, entries=150, sequenceid=654, filesize=11.9 K 2023-04-03 13:58:40,323 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/1f88ad8073194735bcc64e3077e148d6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1f88ad8073194735bcc64e3077e148d6 2023-04-03 13:58:40,325 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1f88ad8073194735bcc64e3077e148d6, entries=150, sequenceid=654, filesize=11.9 K 2023-04-03 13:58:40,326 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a15553c96f61b5a8cc6d1615507e7af8 in 844ms, sequenceid=654, compaction requested=true 2023-04-03 13:58:40,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:40,326 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:40,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:40,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:40,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:40,326 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:40,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,327 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:40,327 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:40,327 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:40,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:40,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:40,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f4f0ff[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:40,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:40,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:40,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:40,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:40,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:40,329 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:40,329 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:40,329 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:40,329 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,329 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:40,329 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:40,329 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:40,329 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,329 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:40,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:40,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:40,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:40,331 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:40,331 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:40,331 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:40,331 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:40,331 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:40,332 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,332 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:40,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:40,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,333 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:40,333 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:40,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:40,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:40,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:40,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:40,343 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:40,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:40,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:40,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:40,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:40,343 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:40,343 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:40,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:40,343 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,443 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:40,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:40,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:40,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:40,446 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:40,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:40,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:40,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:40,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,446 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:40,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,447 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:40,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:40,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:40,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,447 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:40,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:40,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:40,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:40,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:40,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:40,448 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:40,448 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:40,448 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:40,448 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:40,448 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:40,448 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:40,448 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:40,448 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:58:40,448 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:40,449 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:40,449 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:40,449 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:40,449 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:40,449 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:40,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:40,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:40,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530380809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530380810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530380812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,855 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/534ae32b73ac44be9e8135d26bfd2eeb 2023-04-03 13:58:40,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530380915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530380915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:40,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:40,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530380917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530381119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530381119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530381121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,264 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/076b1d490c1b4a08a540bc31be49df3d 2023-04-03 13:58:41,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530381427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530381429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530381430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,672 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/9431a89048804c34aa5092944b84f1e6 2023-04-03 13:58:41,676 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/534ae32b73ac44be9e8135d26bfd2eeb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/534ae32b73ac44be9e8135d26bfd2eeb 2023-04-03 13:58:41,679 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/534ae32b73ac44be9e8135d26bfd2eeb, entries=150, sequenceid=666, filesize=11.9 K 2023-04-03 13:58:41,679 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/076b1d490c1b4a08a540bc31be49df3d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/076b1d490c1b4a08a540bc31be49df3d 2023-04-03 13:58:41,682 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/076b1d490c1b4a08a540bc31be49df3d, entries=150, sequenceid=666, filesize=11.9 K 2023-04-03 13:58:41,683 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/9431a89048804c34aa5092944b84f1e6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9431a89048804c34aa5092944b84f1e6 2023-04-03 13:58:41,685 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9431a89048804c34aa5092944b84f1e6, entries=150, sequenceid=666, filesize=11.9 K 2023-04-03 13:58:41,686 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a15553c96f61b5a8cc6d1615507e7af8 in 1238ms, sequenceid=666, compaction requested=true 2023-04-03 13:58:41,686 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:41,686 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:41,686 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:41,686 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:41,686 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,686 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:41,686 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:41,687 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:41,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:41,687 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:41,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:41,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36d5ad19[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:41,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:41,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:41,690 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:41,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:41,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:41,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:41,690 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,690 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:41,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:41,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:41,691 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:41,691 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,691 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:41,691 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:41,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:41,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:41,692 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:41,693 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:41,693 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:41,693 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:41,693 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:41,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:41,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:41,694 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:41,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:41,695 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:41,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:41,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:41,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:41,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:41,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:41,696 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:41,696 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:41,697 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:41,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:41,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:41,797 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:41,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:41,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:41,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:41,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:41,799 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:41,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:41,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:41,799 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:41,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:41,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,800 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:41,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:41,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,800 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:41,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:41,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:41,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:41,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:41,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:41,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:41,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:41,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:41,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:41,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:41,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:41,801 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:41,801 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:41,801 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:41,801 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 13:58:41,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:41,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:41,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:41,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:41,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:41,802 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:41,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:41,938 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:41,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530381957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530381958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:41,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:41,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530381959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530382064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530382064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530382064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,211 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/a7ccaf6a83e442f4b5180dcbdf7aaabd 2023-04-03 13:58:42,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530382272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530382273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530382273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530382577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530382577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:42,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530382578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,620 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/3c6b56929fdb42829ed2268de822fb98 2023-04-03 13:58:42,628 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/9f4a554ba967470a948719a016172b86 2023-04-03 13:58:42,631 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/a7ccaf6a83e442f4b5180dcbdf7aaabd as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a7ccaf6a83e442f4b5180dcbdf7aaabd 2023-04-03 13:58:42,635 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a7ccaf6a83e442f4b5180dcbdf7aaabd, entries=150, sequenceid=686, filesize=11.9 K 2023-04-03 13:58:42,635 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/3c6b56929fdb42829ed2268de822fb98 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3c6b56929fdb42829ed2268de822fb98 2023-04-03 13:58:42,638 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3c6b56929fdb42829ed2268de822fb98, entries=150, sequenceid=686, filesize=11.9 K 2023-04-03 13:58:42,639 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/9f4a554ba967470a948719a016172b86 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9f4a554ba967470a948719a016172b86 2023-04-03 13:58:42,642 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9f4a554ba967470a948719a016172b86, entries=150, sequenceid=686, filesize=11.9 K 2023-04-03 13:58:42,642 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a15553c96f61b5a8cc6d1615507e7af8 in 841ms, sequenceid=686, compaction requested=true 2023-04-03 13:58:42,642 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:42,642 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:42,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:42,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:42,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:42,642 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:42,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,643 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:42,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:42,644 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:42,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:42,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393d912f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:42,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:42,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:42,645 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:42,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:42,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:42,645 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:42,645 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:42,645 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:42,646 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,646 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:42,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:42,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:42,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:42,647 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:42,647 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:42,647 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:42,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:42,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:42,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:42,648 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:42,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:42,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:42,648 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:42,648 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:42,649 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,650 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:42,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,650 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:42,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:42,650 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:42,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,651 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:42,655 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:42,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:42,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:42,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:42,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:42,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:42,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:42,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:42,655 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,656 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:42,656 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:42,756 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:42,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:42,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:42,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:42,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:42,758 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:42,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:42,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:42,758 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:42,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:42,759 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:42,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:42,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:42,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:42,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:42,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:42,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:42,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:42,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:42,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:42,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:42,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:42,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:42,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:42,761 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:42,761 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:42,761 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:42,761 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:58:42,761 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:42,761 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:42,761 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:42,762 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:42,762 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:42,762 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:43,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:43,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:43,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530383118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530383121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530383123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,172 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/d55e26c5f9a543e89baa984a8b050223 2023-04-03 13:58:43,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530383223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530383225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530383230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530383430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530383430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530383439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,580 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/077f9fd02381465ca57775fdd453a520 2023-04-03 13:58:43,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530383737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530383737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:43,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530383744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:43,988 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/2c828cc9a4384175b8a8946d8298066e 2023-04-03 13:58:43,991 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/d55e26c5f9a543e89baa984a8b050223 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/d55e26c5f9a543e89baa984a8b050223 2023-04-03 13:58:43,994 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/d55e26c5f9a543e89baa984a8b050223, entries=100, sequenceid=698, filesize=9.5 K 2023-04-03 13:58:43,995 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/077f9fd02381465ca57775fdd453a520 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/077f9fd02381465ca57775fdd453a520 2023-04-03 13:58:43,998 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/077f9fd02381465ca57775fdd453a520, entries=100, sequenceid=698, filesize=9.5 K 2023-04-03 13:58:43,998 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/2c828cc9a4384175b8a8946d8298066e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/2c828cc9a4384175b8a8946d8298066e 2023-04-03 13:58:44,001 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/2c828cc9a4384175b8a8946d8298066e, entries=100, sequenceid=698, filesize=9.5 K 2023-04-03 13:58:44,002 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a15553c96f61b5a8cc6d1615507e7af8 in 1241ms, sequenceid=698, compaction requested=true 2023-04-03 13:58:44,002 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:44,002 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:44,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:44,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:44,003 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,003 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:44,003 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:44,003 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:44,004 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:44,004 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,004 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:44,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:44,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b1dfef[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:44,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:44,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,005 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:44,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:44,005 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:44,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:44,006 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:44,006 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,006 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:44,006 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:44,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:44,007 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:44,007 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:44,007 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:44,007 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:44,007 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:44,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:44,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,008 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,009 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:44,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:44,010 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:44,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:44,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:44,010 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:44,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:44,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,011 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:44,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:44,011 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:44,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:44,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:44,111 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:44,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:44,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:44,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:44,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:44,115 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:44,115 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:44,115 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:44,115 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:44,115 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,115 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:44,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:44,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:44,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:44,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:44,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:44,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:44,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:44,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:44,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:44,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:44,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:44,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:44,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:44,117 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:44,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:44,117 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:44,117 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 13:58:44,118 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:44,118 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:44,118 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:44,118 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:44,118 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:44,118 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:44,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:44,250 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:44,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530384261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530384273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530384274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530384375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530384376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,388 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530384383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,529 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/5a1fc2ad93454953b26ea521626b0144 2023-04-03 13:58:44,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530384581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530384584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530384590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40534 deadline: 1680530384770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,774 DEBUG [Thread-2877] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18206 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:58:44,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:40578 deadline: 1680530384771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,780 DEBUG [Thread-2873] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18209 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:58:44,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530384887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530384892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:44,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530384896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,939 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/7eb7e8dac71f4dcc9825b03026136a40 2023-04-03 13:58:44,956 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/18b43f0e80e6455fab9e6bd2bc6c7b7c 2023-04-03 13:58:44,960 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/5a1fc2ad93454953b26ea521626b0144 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/5a1fc2ad93454953b26ea521626b0144 2023-04-03 13:58:44,962 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/5a1fc2ad93454953b26ea521626b0144, entries=150, sequenceid=718, filesize=11.9 K 2023-04-03 13:58:44,963 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/7eb7e8dac71f4dcc9825b03026136a40 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/7eb7e8dac71f4dcc9825b03026136a40 2023-04-03 13:58:44,966 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/7eb7e8dac71f4dcc9825b03026136a40, entries=150, sequenceid=718, filesize=11.9 K 2023-04-03 13:58:44,967 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/18b43f0e80e6455fab9e6bd2bc6c7b7c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/18b43f0e80e6455fab9e6bd2bc6c7b7c 2023-04-03 13:58:44,969 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/18b43f0e80e6455fab9e6bd2bc6c7b7c, entries=150, sequenceid=718, filesize=11.9 K 2023-04-03 13:58:44,970 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a15553c96f61b5a8cc6d1615507e7af8 in 853ms, sequenceid=718, compaction requested=true 2023-04-03 13:58:44,970 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:44,970 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:44,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:44,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:44,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:44,970 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:44,971 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:44,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,971 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:44,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:44,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f0a2ce[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:44,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:44,974 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,975 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:44,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:44,975 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,975 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:44,975 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:44,976 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,976 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:44,976 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:44,976 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:44,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:44,978 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:44,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:44,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:44,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:44,978 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:44,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,978 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:44,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,979 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:44,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:44,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,982 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:44,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:44,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:44,987 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:44,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:44,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:44,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:44,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:44,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:44,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:44,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:44,988 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:44,988 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:44,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:44,988 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:45,088 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:45,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:45,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:45,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:45,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:45,091 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:45,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:45,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:45,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:45,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:45,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:45,092 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:45,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:45,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,092 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:45,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:45,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:45,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:45,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:45,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:45,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:45,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:45,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:45,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:45,093 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:45,094 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:45,094 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. started... 2023-04-03 13:58:45,094 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:58:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:45,094 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:45,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:45,403 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. as already flushing 2023-04-03 13:58:45,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530385462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530385463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530385464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,501 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/641be0e73a73460197cb5c3ca074d4b2 2023-04-03 13:58:45,510 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/78af9c673ee6461fa19c96600940d5aa 2023-04-03 13:58:45,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530385579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530385579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530385579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,772 DEBUG [Thread-2886] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ea9d51a to 127.0.0.1:57081 2023-04-03 13:58:45,772 DEBUG [Thread-2886] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:45,772 DEBUG [Thread-2888] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fd1c1f to 127.0.0.1:57081 2023-04-03 13:58:45,772 DEBUG [Thread-2888] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:45,775 DEBUG [Thread-2890] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x209faa3c to 127.0.0.1:57081 2023-04-03 13:58:45,775 DEBUG [Thread-2890] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:45,780 DEBUG [Thread-2884] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5d7fcf to 127.0.0.1:57081 2023-04-03 13:58:45,780 DEBUG [Thread-2884] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:45,782 DEBUG [Thread-2882] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e6f624 to 127.0.0.1:57081 2023-04-03 13:58:45,782 DEBUG [Thread-2882] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:45,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40582 deadline: 1680530385788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40596 deadline: 1680530385788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:45,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:40580 deadline: 1680530385789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,918 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=730 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/6c2c148a30f444e395bf302679765f23 2023-04-03 13:58:45,927 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/641be0e73a73460197cb5c3ca074d4b2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/641be0e73a73460197cb5c3ca074d4b2 2023-04-03 13:58:45,932 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/641be0e73a73460197cb5c3ca074d4b2, entries=150, sequenceid=730, filesize=11.9 K 2023-04-03 13:58:45,933 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/78af9c673ee6461fa19c96600940d5aa as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/78af9c673ee6461fa19c96600940d5aa 2023-04-03 13:58:45,938 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/78af9c673ee6461fa19c96600940d5aa, entries=150, sequenceid=730, filesize=11.9 K 2023-04-03 13:58:45,939 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/6c2c148a30f444e395bf302679765f23 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c2c148a30f444e395bf302679765f23 2023-04-03 13:58:45,944 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c2c148a30f444e395bf302679765f23, entries=150, sequenceid=730, filesize=11.9 K 2023-04-03 13:58:45,945 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a15553c96f61b5a8cc6d1615507e7af8 in 851ms, sequenceid=730, compaction requested=true 2023-04-03 13:58:45,945 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:45,945 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:45,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:45,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:45,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:45,945 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:45,947 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,947 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:45,948 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:45,948 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:45,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:45,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd67a2c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:45,950 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:45,950 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:45,951 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:45,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:45,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:45,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,951 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:45,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:45,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:45,952 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:45,953 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:45,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:45,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:45,956 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:45,956 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:45,956 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:45,956 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:45,957 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:45,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:45,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:45,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:45,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:45,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:45,960 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:45,962 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,969 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,969 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:45,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:45,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:58:45,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:45,969 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:45,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:45,969 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:45,969 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:45,969 DEBUG [Thread-2881] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:45,969 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:45,970 DEBUG [Thread-2881] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:45,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:45,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:45,970 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:45,970 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:45,970 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:45,970 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:46,070 DEBUG [Thread-2881] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:46,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:46,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:46,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-03 13:58:46,091 DEBUG [Thread-2871] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e895a5d to 127.0.0.1:57081 2023-04-03 13:58:46,091 DEBUG [Thread-2871] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:46,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:46,092 DEBUG [Thread-2879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x647f95f6 to 127.0.0.1:57081 2023-04-03 13:58:46,092 DEBUG [Thread-2879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:46,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:46,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:46,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:46,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:46,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:46,093 DEBUG [Thread-2875] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ff7cd4 to 127.0.0.1:57081 2023-04-03 13:58:46,093 DEBUG [Thread-2875] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:46,498 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/150e3c68b0a34896bbc7a4284227326e 2023-04-03 13:58:46,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/ad1f800fc9a34c01813b12e859048ad3 2023-04-03 13:58:47,329 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/704a95ff8b69482d8c7c5aadcfb4ae2a 2023-04-03 13:58:47,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/150e3c68b0a34896bbc7a4284227326e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/150e3c68b0a34896bbc7a4284227326e 2023-04-03 13:58:47,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/150e3c68b0a34896bbc7a4284227326e, entries=150, sequenceid=752, filesize=11.9 K 2023-04-03 13:58:47,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/ad1f800fc9a34c01813b12e859048ad3 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ad1f800fc9a34c01813b12e859048ad3 2023-04-03 13:58:47,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ad1f800fc9a34c01813b12e859048ad3, entries=150, sequenceid=752, filesize=11.9 K 2023-04-03 13:58:47,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/704a95ff8b69482d8c7c5aadcfb4ae2a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/704a95ff8b69482d8c7c5aadcfb4ae2a 2023-04-03 13:58:47,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/704a95ff8b69482d8c7c5aadcfb4ae2a, entries=150, sequenceid=752, filesize=11.9 K 2023-04-03 13:58:47,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=6.71 KB/6870 for a15553c96f61b5a8cc6d1615507e7af8 in 1267ms, sequenceid=752, compaction requested=true 2023-04-03 13:58:47,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:47,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:47,358 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 13:58:47,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:47,358 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 13:58:47,358 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:47,361 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119894 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 13:58:47,362 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/A is initiating minor compaction (all files) 2023-04-03 13:58:47,362 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119894 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 13:58:47,362 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/A in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:47,362 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/B is initiating minor compaction (all files) 2023-04-03 13:58:47,362 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/9f3d474ba77a43819f8cd279e7f42cf2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/74504975269841ebb8faa51d1fd58e38, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a70bf8abf1924b34969d47566c5e66b2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/af73738c38c74aff90624facd9f6ef00, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/534ae32b73ac44be9e8135d26bfd2eeb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a7ccaf6a83e442f4b5180dcbdf7aaabd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/d55e26c5f9a543e89baa984a8b050223, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/5a1fc2ad93454953b26ea521626b0144, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/641be0e73a73460197cb5c3ca074d4b2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/150e3c68b0a34896bbc7a4284227326e] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=117.1 K 2023-04-03 13:58:47,362 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/B in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:47,362 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c4731d40a99349d1aefbd91037d2fe40, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/19e7ac9aca54420592cdf915b4d6e746, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/bb53b3333e75400ca62cf810186c75c5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c727c4c425944b08a65f49b8a1dcfbf9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/076b1d490c1b4a08a540bc31be49df3d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3c6b56929fdb42829ed2268de822fb98, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/077f9fd02381465ca57775fdd453a520, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/7eb7e8dac71f4dcc9825b03026136a40, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/78af9c673ee6461fa19c96600940d5aa, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ad1f800fc9a34c01813b12e859048ad3] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=117.1 K 2023-04-03 13:58:47,362 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9f3d474ba77a43819f8cd279e7f42cf2, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680530316681 2023-04-03 13:58:47,363 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c4731d40a99349d1aefbd91037d2fe40, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680530316681 2023-04-03 13:58:47,363 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 74504975269841ebb8faa51d1fd58e38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680530316748 2023-04-03 13:58:47,363 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 19e7ac9aca54420592cdf915b4d6e746, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680530316748 2023-04-03 13:58:47,363 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting a70bf8abf1924b34969d47566c5e66b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680530317889 2023-04-03 13:58:47,364 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bb53b3333e75400ca62cf810186c75c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680530317889 2023-04-03 13:58:47,364 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting af73738c38c74aff90624facd9f6ef00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680530318532 2023-04-03 13:58:47,364 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c727c4c425944b08a65f49b8a1dcfbf9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680530318532 2023-04-03 13:58:47,364 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 534ae32b73ac44be9e8135d26bfd2eeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680530319654 2023-04-03 13:58:47,364 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 076b1d490c1b4a08a540bc31be49df3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680530319654 2023-04-03 13:58:47,364 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting a7ccaf6a83e442f4b5180dcbdf7aaabd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1680530320795 2023-04-03 13:58:47,365 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 3c6b56929fdb42829ed2268de822fb98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1680530320795 2023-04-03 13:58:47,365 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting d55e26c5f9a543e89baa984a8b050223, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680530321956 2023-04-03 13:58:47,365 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 077f9fd02381465ca57775fdd453a520, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680530321956 2023-04-03 13:58:47,365 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 5a1fc2ad93454953b26ea521626b0144, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680530323116 2023-04-03 13:58:47,365 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 7eb7e8dac71f4dcc9825b03026136a40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680530323116 2023-04-03 13:58:47,366 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 641be0e73a73460197cb5c3ca074d4b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=730, earliestPutTs=1680530324259 2023-04-03 13:58:47,366 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 78af9c673ee6461fa19c96600940d5aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=730, earliestPutTs=1680530324259 2023-04-03 13:58:47,366 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 150e3c68b0a34896bbc7a4284227326e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1680530325438 2023-04-03 13:58:47,366 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ad1f800fc9a34c01813b12e859048ad3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1680530325438 2023-04-03 13:58:47,384 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#A#compaction#435 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:47,384 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#B#compaction#436 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:47,390 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/99d25b74f7964f428a445523f216a68d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99d25b74f7964f428a445523f216a68d 2023-04-03 13:58:47,393 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/A of a15553c96f61b5a8cc6d1615507e7af8 into 99d25b74f7964f428a445523f216a68d(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:47,393 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:47,393 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/A, priority=6, startTime=1680530327358; duration=0sec 2023-04-03 13:58:47,393 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:58:47,394 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 13:58:47,395 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119894 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 13:58:47,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): a15553c96f61b5a8cc6d1615507e7af8/C is initiating minor compaction (all files) 2023-04-03 13:58:47,396 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a15553c96f61b5a8cc6d1615507e7af8/C in TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:47,396 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1a981f724f8d4a69a4475b20c2d4658a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1e340b0542b5433982c73da1aefe5cac, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e5357c4370d424fbf688ca7ab9bb042, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1f88ad8073194735bcc64e3077e148d6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9431a89048804c34aa5092944b84f1e6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9f4a554ba967470a948719a016172b86, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/2c828cc9a4384175b8a8946d8298066e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/18b43f0e80e6455fab9e6bd2bc6c7b7c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c2c148a30f444e395bf302679765f23, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/704a95ff8b69482d8c7c5aadcfb4ae2a] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp, totalSize=117.1 K 2023-04-03 13:58:47,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 1a981f724f8d4a69a4475b20c2d4658a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680530316681 2023-04-03 13:58:47,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 1e340b0542b5433982c73da1aefe5cac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=620, earliestPutTs=1680530316748 2023-04-03 13:58:47,396 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 3e5357c4370d424fbf688ca7ab9bb042, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=634, earliestPutTs=1680530317889 2023-04-03 13:58:47,397 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 1f88ad8073194735bcc64e3077e148d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680530318532 2023-04-03 13:58:47,397 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9431a89048804c34aa5092944b84f1e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680530319654 2023-04-03 13:58:47,397 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 9f4a554ba967470a948719a016172b86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=686, earliestPutTs=1680530320795 2023-04-03 13:58:47,397 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 2c828cc9a4384175b8a8946d8298066e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680530321956 2023-04-03 13:58:47,397 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 18b43f0e80e6455fab9e6bd2bc6c7b7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680530323116 2023-04-03 13:58:47,397 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 6c2c148a30f444e395bf302679765f23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=730, earliestPutTs=1680530324259 2023-04-03 13:58:47,398 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 704a95ff8b69482d8c7c5aadcfb4ae2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=752, earliestPutTs=1680530325438 2023-04-03 13:58:47,412 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): a15553c96f61b5a8cc6d1615507e7af8#C#compaction#437 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:58:47,420 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/fa5e8f2994f446249ae8bc47582b27d3 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/fa5e8f2994f446249ae8bc47582b27d3 2023-04-03 13:58:47,425 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/C of a15553c96f61b5a8cc6d1615507e7af8 into fa5e8f2994f446249ae8bc47582b27d3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:47,425 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:47,425 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/C, priority=6, startTime=1680530327358; duration=0sec 2023-04-03 13:58:47,425 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:47,793 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/e7e13eb0f1c44c3eaa026aaea22aa330 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e7e13eb0f1c44c3eaa026aaea22aa330 2023-04-03 13:58:47,800 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in a15553c96f61b5a8cc6d1615507e7af8/B of a15553c96f61b5a8cc6d1615507e7af8 into e7e13eb0f1c44c3eaa026aaea22aa330(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:58:47,800 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:47,800 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8., storeName=a15553c96f61b5a8cc6d1615507e7af8/B, priority=6, startTime=1680530327358; duration=0sec 2023-04-03 13:58:47,801 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:58:54,785 DEBUG [Thread-2873] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ea7ac79 to 127.0.0.1:57081 2023-04-03 13:58:54,785 DEBUG [Thread-2873] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:54,860 DEBUG [Thread-2877] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06846250 to 127.0.0.1:57081 2023-04-03 13:58:54,860 DEBUG [Thread-2877] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:54,860 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 13:58:54,861 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 170 2023-04-03 13:58:54,861 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-04-03 13:58:54,861 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 219 2023-04-03 13:58:54,862 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-03 13:58:54,862 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-03 13:58:54,862 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 13:58:54,863 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 13:58:54,863 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2168 2023-04-03 13:58:54,863 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 6504 rows 2023-04-03 13:58:54,864 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2182 2023-04-03 13:58:54,864 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 6544 rows 2023-04-03 13:58:54,864 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2170 2023-04-03 13:58:54,864 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 6508 rows 2023-04-03 13:58:54,865 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2177 2023-04-03 13:58:54,865 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 6529 rows 2023-04-03 13:58:54,865 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(402): scanned 2167 2023-04-03 13:58:54,866 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(403): verified 6499 rows 2023-04-03 13:58:54,866 INFO [Listener at localhost.localdomain/43477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 13:58:54,866 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x517cd407 to 127.0.0.1:57081 2023-04-03 13:58:54,866 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:58:54,867 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 13:58:54,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-03 13:58:54,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:54,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 13:58:54,871 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530334871"}]},"ts":"1680530334871"} 2023-04-03 13:58:54,872 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 13:58:54,873 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 13:58:54,873 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a15553c96f61b5a8cc6d1615507e7af8, UNASSIGN}] 2023-04-03 13:58:54,874 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a15553c96f61b5a8cc6d1615507e7af8, UNASSIGN 2023-04-03 13:58:54,875 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=a15553c96f61b5a8cc6d1615507e7af8, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:54,875 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530334875"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530334875"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530334875"}]},"ts":"1680530334875"} 2023-04-03 13:58:54,876 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:58:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 13:58:55,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:55,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing a15553c96f61b5a8cc6d1615507e7af8, disabling compactions & flushes 2023-04-03 13:58:55,028 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:55,028 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:55,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. after waiting 0 ms 2023-04-03 13:58:55,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:55,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing a15553c96f61b5a8cc6d1615507e7af8 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-03 13:58:55,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=A 2023-04-03 13:58:55,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:55,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=B 2023-04-03 13:58:55,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:55,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a15553c96f61b5a8cc6d1615507e7af8, store=C 2023-04-03 13:58:55,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:55,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 13:58:55,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/afa8edad473e4cdf8e2404df76129b66 2023-04-03 13:58:55,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/e31f7166eaef4c9ab91b89084604e7e4 2023-04-03 13:58:55,469 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=761 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/e430b9f47dda4c1facff5e20286357c8 2023-04-03 13:58:55,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/A/afa8edad473e4cdf8e2404df76129b66 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/afa8edad473e4cdf8e2404df76129b66 2023-04-03 13:58:55,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 13:58:55,479 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/afa8edad473e4cdf8e2404df76129b66, entries=150, sequenceid=761, filesize=11.9 K 2023-04-03 13:58:55,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/B/e31f7166eaef4c9ab91b89084604e7e4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e31f7166eaef4c9ab91b89084604e7e4 2023-04-03 13:58:55,483 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e31f7166eaef4c9ab91b89084604e7e4, entries=150, sequenceid=761, filesize=11.9 K 2023-04-03 13:58:55,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/.tmp/C/e430b9f47dda4c1facff5e20286357c8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/e430b9f47dda4c1facff5e20286357c8 2023-04-03 13:58:55,487 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/e430b9f47dda4c1facff5e20286357c8, entries=150, sequenceid=761, filesize=11.9 K 2023-04-03 13:58:55,488 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for a15553c96f61b5a8cc6d1615507e7af8 in 459ms, sequenceid=761, compaction requested=false 2023-04-03 13:58:55,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/09184d412adf46589e539756170752e4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2f0b77d287ea4f08b370a9f5a4d66a45, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/add52675500e45018f776e9c174dd361, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/e77bb656c2974fef8ca6ed43b0f0f724, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/43517bfd6e3541a58716c466dd65b31a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99940e22a3fa4ec1879a5018770020e6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2343d435eec840e8b3aff836824d8b3e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/1b9b07bd589a462f8bb9522c45fd4af6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/179688ebc78d469cae381fd71fe7cff5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/0257d3b23dd34e0d9cff3be22f2f2bd2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/59fe5a7924d148d1a85c82fc78ba170f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/7a39c390590a44aa8911f126c91fd5c4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/063a5a8d8e1b47c69dda0b0b8371f8a0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/b237e0171d88476c928aacdba4777c73, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/9f3d474ba77a43819f8cd279e7f42cf2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/74504975269841ebb8faa51d1fd58e38, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a70bf8abf1924b34969d47566c5e66b2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/af73738c38c74aff90624facd9f6ef00, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/534ae32b73ac44be9e8135d26bfd2eeb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a7ccaf6a83e442f4b5180dcbdf7aaabd, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/d55e26c5f9a543e89baa984a8b050223, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/5a1fc2ad93454953b26ea521626b0144, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/641be0e73a73460197cb5c3ca074d4b2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/150e3c68b0a34896bbc7a4284227326e] to archive 2023-04-03 13:58:55,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:55,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/09184d412adf46589e539756170752e4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/09184d412adf46589e539756170752e4 2023-04-03 13:58:55,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2f0b77d287ea4f08b370a9f5a4d66a45 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2f0b77d287ea4f08b370a9f5a4d66a45 2023-04-03 13:58:55,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/add52675500e45018f776e9c174dd361 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/add52675500e45018f776e9c174dd361 2023-04-03 13:58:55,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/e77bb656c2974fef8ca6ed43b0f0f724 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/e77bb656c2974fef8ca6ed43b0f0f724 2023-04-03 13:58:55,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/43517bfd6e3541a58716c466dd65b31a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/43517bfd6e3541a58716c466dd65b31a 2023-04-03 13:58:55,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99940e22a3fa4ec1879a5018770020e6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99940e22a3fa4ec1879a5018770020e6 2023-04-03 13:58:55,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2343d435eec840e8b3aff836824d8b3e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/2343d435eec840e8b3aff836824d8b3e 2023-04-03 13:58:55,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/1b9b07bd589a462f8bb9522c45fd4af6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/1b9b07bd589a462f8bb9522c45fd4af6 2023-04-03 13:58:55,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/179688ebc78d469cae381fd71fe7cff5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/179688ebc78d469cae381fd71fe7cff5 2023-04-03 13:58:55,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/0257d3b23dd34e0d9cff3be22f2f2bd2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/0257d3b23dd34e0d9cff3be22f2f2bd2 2023-04-03 13:58:55,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/59fe5a7924d148d1a85c82fc78ba170f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/59fe5a7924d148d1a85c82fc78ba170f 2023-04-03 13:58:55,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/7a39c390590a44aa8911f126c91fd5c4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/7a39c390590a44aa8911f126c91fd5c4 2023-04-03 13:58:55,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/063a5a8d8e1b47c69dda0b0b8371f8a0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/063a5a8d8e1b47c69dda0b0b8371f8a0 2023-04-03 13:58:55,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/b237e0171d88476c928aacdba4777c73 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/b237e0171d88476c928aacdba4777c73 2023-04-03 13:58:55,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/9f3d474ba77a43819f8cd279e7f42cf2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/9f3d474ba77a43819f8cd279e7f42cf2 2023-04-03 13:58:55,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/74504975269841ebb8faa51d1fd58e38 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/74504975269841ebb8faa51d1fd58e38 2023-04-03 13:58:55,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a70bf8abf1924b34969d47566c5e66b2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a70bf8abf1924b34969d47566c5e66b2 2023-04-03 13:58:55,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/af73738c38c74aff90624facd9f6ef00 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/af73738c38c74aff90624facd9f6ef00 2023-04-03 13:58:55,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/534ae32b73ac44be9e8135d26bfd2eeb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/534ae32b73ac44be9e8135d26bfd2eeb 2023-04-03 13:58:55,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a7ccaf6a83e442f4b5180dcbdf7aaabd to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/a7ccaf6a83e442f4b5180dcbdf7aaabd 2023-04-03 13:58:55,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/d55e26c5f9a543e89baa984a8b050223 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/d55e26c5f9a543e89baa984a8b050223 2023-04-03 13:58:55,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/5a1fc2ad93454953b26ea521626b0144 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/5a1fc2ad93454953b26ea521626b0144 2023-04-03 13:58:55,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/641be0e73a73460197cb5c3ca074d4b2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/641be0e73a73460197cb5c3ca074d4b2 2023-04-03 13:58:55,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/150e3c68b0a34896bbc7a4284227326e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/150e3c68b0a34896bbc7a4284227326e 2023-04-03 13:58:55,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4eda35df51a147b898bc667aa51e1385, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a4aea3ee3f7d48d8bd40a42c3e2fa702, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/8b1c06dc407b429a9876bce40f5004bf, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ab1319f57cb24d88922a46de6b420117, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/9884099e2ad54120883a00931b27d510, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/6f6012fca4c64fc297b7491896c7fa34, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3deb81a607644bda9353bf479a92532b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a9f85332739c479bb2d09d4d334e6e91, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e2f2462c36184d92a4072a98443bed58, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/f75a35d24d1045359a0f0428f202d55b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/268321b0193440609f14178af1f312e7, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4380124443db4b7dbdc2523af6dfaf4a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/2a08dfbceabf47b890e6a45212e1dde9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c4731d40a99349d1aefbd91037d2fe40, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/33b0e0ccfaca47f3bfa20a08b7d8d8be, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/19e7ac9aca54420592cdf915b4d6e746, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/bb53b3333e75400ca62cf810186c75c5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c727c4c425944b08a65f49b8a1dcfbf9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/076b1d490c1b4a08a540bc31be49df3d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3c6b56929fdb42829ed2268de822fb98, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/077f9fd02381465ca57775fdd453a520, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/7eb7e8dac71f4dcc9825b03026136a40, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/78af9c673ee6461fa19c96600940d5aa, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ad1f800fc9a34c01813b12e859048ad3] to archive 2023-04-03 13:58:55,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:55,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4eda35df51a147b898bc667aa51e1385 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4eda35df51a147b898bc667aa51e1385 2023-04-03 13:58:55,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a4aea3ee3f7d48d8bd40a42c3e2fa702 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a4aea3ee3f7d48d8bd40a42c3e2fa702 2023-04-03 13:58:55,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/8b1c06dc407b429a9876bce40f5004bf to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/8b1c06dc407b429a9876bce40f5004bf 2023-04-03 13:58:55,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ab1319f57cb24d88922a46de6b420117 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ab1319f57cb24d88922a46de6b420117 2023-04-03 13:58:55,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/9884099e2ad54120883a00931b27d510 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/9884099e2ad54120883a00931b27d510 2023-04-03 13:58:55,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/6f6012fca4c64fc297b7491896c7fa34 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/6f6012fca4c64fc297b7491896c7fa34 2023-04-03 13:58:55,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3deb81a607644bda9353bf479a92532b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3deb81a607644bda9353bf479a92532b 2023-04-03 13:58:55,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a9f85332739c479bb2d09d4d334e6e91 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/a9f85332739c479bb2d09d4d334e6e91 2023-04-03 13:58:55,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e2f2462c36184d92a4072a98443bed58 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e2f2462c36184d92a4072a98443bed58 2023-04-03 13:58:55,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/f75a35d24d1045359a0f0428f202d55b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/f75a35d24d1045359a0f0428f202d55b 2023-04-03 13:58:55,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/268321b0193440609f14178af1f312e7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/268321b0193440609f14178af1f312e7 2023-04-03 13:58:55,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4380124443db4b7dbdc2523af6dfaf4a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/4380124443db4b7dbdc2523af6dfaf4a 2023-04-03 13:58:55,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/2a08dfbceabf47b890e6a45212e1dde9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/2a08dfbceabf47b890e6a45212e1dde9 2023-04-03 13:58:55,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c4731d40a99349d1aefbd91037d2fe40 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c4731d40a99349d1aefbd91037d2fe40 2023-04-03 13:58:55,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/33b0e0ccfaca47f3bfa20a08b7d8d8be to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/33b0e0ccfaca47f3bfa20a08b7d8d8be 2023-04-03 13:58:55,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/19e7ac9aca54420592cdf915b4d6e746 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/19e7ac9aca54420592cdf915b4d6e746 2023-04-03 13:58:55,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/bb53b3333e75400ca62cf810186c75c5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/bb53b3333e75400ca62cf810186c75c5 2023-04-03 13:58:55,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c727c4c425944b08a65f49b8a1dcfbf9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/c727c4c425944b08a65f49b8a1dcfbf9 2023-04-03 13:58:55,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/076b1d490c1b4a08a540bc31be49df3d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/076b1d490c1b4a08a540bc31be49df3d 2023-04-03 13:58:55,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3c6b56929fdb42829ed2268de822fb98 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/3c6b56929fdb42829ed2268de822fb98 2023-04-03 13:58:55,530 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/077f9fd02381465ca57775fdd453a520 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/077f9fd02381465ca57775fdd453a520 2023-04-03 13:58:55,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/7eb7e8dac71f4dcc9825b03026136a40 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/7eb7e8dac71f4dcc9825b03026136a40 2023-04-03 13:58:55,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/78af9c673ee6461fa19c96600940d5aa to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/78af9c673ee6461fa19c96600940d5aa 2023-04-03 13:58:55,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ad1f800fc9a34c01813b12e859048ad3 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/ad1f800fc9a34c01813b12e859048ad3 2023-04-03 13:58:55,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/b16fcce3455a4cfebee06384d872e8c5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/24ac3348f482421993d0ef4114ef2456, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e890cb90ebd4b1d8c505503e2a6321a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/d633b3c650bd461f85f4cf82b980ec33, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/bed4efc7009f4a44a9b2a06ea50c9a64, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c97ce5473a741d9ac5576f2792dc60d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5cd8410dc2714fac8fb0665cde23c760, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/fda9d49b34254537976052d5d3532969, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/734f09b0b04249a4b18c758675f0a6ac, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5743cb4c30bd4a8794d548ad311e7db4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/941baa34f5704c2a9b337d6f9261da15, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/23aa35df157f478d970cc80c89e2031e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5b08160a8a4b4d25932afda5ac261917, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1a981f724f8d4a69a4475b20c2d4658a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/ada5ac3b58794069867d6f38ea741184, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1e340b0542b5433982c73da1aefe5cac, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e5357c4370d424fbf688ca7ab9bb042, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1f88ad8073194735bcc64e3077e148d6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9431a89048804c34aa5092944b84f1e6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9f4a554ba967470a948719a016172b86, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/2c828cc9a4384175b8a8946d8298066e, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/18b43f0e80e6455fab9e6bd2bc6c7b7c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c2c148a30f444e395bf302679765f23, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/704a95ff8b69482d8c7c5aadcfb4ae2a] to archive 2023-04-03 13:58:55,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:58:55,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/b16fcce3455a4cfebee06384d872e8c5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/b16fcce3455a4cfebee06384d872e8c5 2023-04-03 13:58:55,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/24ac3348f482421993d0ef4114ef2456 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/24ac3348f482421993d0ef4114ef2456 2023-04-03 13:58:55,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e890cb90ebd4b1d8c505503e2a6321a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e890cb90ebd4b1d8c505503e2a6321a 2023-04-03 13:58:55,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/d633b3c650bd461f85f4cf82b980ec33 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/d633b3c650bd461f85f4cf82b980ec33 2023-04-03 13:58:55,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/bed4efc7009f4a44a9b2a06ea50c9a64 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/bed4efc7009f4a44a9b2a06ea50c9a64 2023-04-03 13:58:55,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c97ce5473a741d9ac5576f2792dc60d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c97ce5473a741d9ac5576f2792dc60d 2023-04-03 13:58:55,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5cd8410dc2714fac8fb0665cde23c760 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5cd8410dc2714fac8fb0665cde23c760 2023-04-03 13:58:55,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/fda9d49b34254537976052d5d3532969 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/fda9d49b34254537976052d5d3532969 2023-04-03 13:58:55,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/734f09b0b04249a4b18c758675f0a6ac to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/734f09b0b04249a4b18c758675f0a6ac 2023-04-03 13:58:55,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5743cb4c30bd4a8794d548ad311e7db4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5743cb4c30bd4a8794d548ad311e7db4 2023-04-03 13:58:55,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/941baa34f5704c2a9b337d6f9261da15 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/941baa34f5704c2a9b337d6f9261da15 2023-04-03 13:58:55,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/23aa35df157f478d970cc80c89e2031e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/23aa35df157f478d970cc80c89e2031e 2023-04-03 13:58:55,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5b08160a8a4b4d25932afda5ac261917 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/5b08160a8a4b4d25932afda5ac261917 2023-04-03 13:58:55,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1a981f724f8d4a69a4475b20c2d4658a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1a981f724f8d4a69a4475b20c2d4658a 2023-04-03 13:58:55,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/ada5ac3b58794069867d6f38ea741184 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/ada5ac3b58794069867d6f38ea741184 2023-04-03 13:58:55,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1e340b0542b5433982c73da1aefe5cac to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1e340b0542b5433982c73da1aefe5cac 2023-04-03 13:58:55,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e5357c4370d424fbf688ca7ab9bb042 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/3e5357c4370d424fbf688ca7ab9bb042 2023-04-03 13:58:55,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1f88ad8073194735bcc64e3077e148d6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/1f88ad8073194735bcc64e3077e148d6 2023-04-03 13:58:55,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9431a89048804c34aa5092944b84f1e6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9431a89048804c34aa5092944b84f1e6 2023-04-03 13:58:55,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9f4a554ba967470a948719a016172b86 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/9f4a554ba967470a948719a016172b86 2023-04-03 13:58:55,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/2c828cc9a4384175b8a8946d8298066e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/2c828cc9a4384175b8a8946d8298066e 2023-04-03 13:58:55,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/18b43f0e80e6455fab9e6bd2bc6c7b7c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/18b43f0e80e6455fab9e6bd2bc6c7b7c 2023-04-03 13:58:55,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c2c148a30f444e395bf302679765f23 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/6c2c148a30f444e395bf302679765f23 2023-04-03 13:58:55,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/704a95ff8b69482d8c7c5aadcfb4ae2a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/704a95ff8b69482d8c7c5aadcfb4ae2a 2023-04-03 13:58:55,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/recovered.edits/764.seqid, newMaxSeqId=764, maxSeqId=1 2023-04-03 13:58:55,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8. 2023-04-03 13:58:55,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for a15553c96f61b5a8cc6d1615507e7af8: 2023-04-03 13:58:55,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:55,563 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=a15553c96f61b5a8cc6d1615507e7af8, regionState=CLOSED 2023-04-03 13:58:55,563 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530335563"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530335563"}]},"ts":"1680530335563"} 2023-04-03 13:58:55,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-03 13:58:55,564 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure a15553c96f61b5a8cc6d1615507e7af8, server=jenkins-hbase20.apache.org,39695,1680530174856 in 687 msec 2023-04-03 13:58:55,565 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-03 13:58:55,565 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a15553c96f61b5a8cc6d1615507e7af8, UNASSIGN in 691 msec 2023-04-03 13:58:55,566 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530335566"}]},"ts":"1680530335566"} 2023-04-03 13:58:55,566 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 13:58:55,567 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 13:58:55,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 700 msec 2023-04-03 13:58:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 13:58:55,977 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-03 13:58:55,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-03 13:58:55,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:55,983 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:55,984 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:55,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-03 13:58:55,992 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:55,996 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/recovered.edits] 2023-04-03 13:58:56,001 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99d25b74f7964f428a445523f216a68d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/99d25b74f7964f428a445523f216a68d 2023-04-03 13:58:56,003 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/afa8edad473e4cdf8e2404df76129b66 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/A/afa8edad473e4cdf8e2404df76129b66 2023-04-03 13:58:56,006 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e31f7166eaef4c9ab91b89084604e7e4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e31f7166eaef4c9ab91b89084604e7e4 2023-04-03 13:58:56,008 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e7e13eb0f1c44c3eaa026aaea22aa330 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/B/e7e13eb0f1c44c3eaa026aaea22aa330 2023-04-03 13:58:56,012 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/e430b9f47dda4c1facff5e20286357c8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/e430b9f47dda4c1facff5e20286357c8 2023-04-03 13:58:56,014 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/fa5e8f2994f446249ae8bc47582b27d3 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/C/fa5e8f2994f446249ae8bc47582b27d3 2023-04-03 13:58:56,018 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/recovered.edits/764.seqid to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8/recovered.edits/764.seqid 2023-04-03 13:58:56,019 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/a15553c96f61b5a8cc6d1615507e7af8 2023-04-03 13:58:56,019 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:58:56,023 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:56,025 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 13:58:56,027 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 13:58:56,029 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:56,029 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 13:58:56,029 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680530336029"}]},"ts":"9223372036854775807"} 2023-04-03 13:58:56,032 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 13:58:56,032 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a15553c96f61b5a8cc6d1615507e7af8, NAME => 'TestAcidGuarantees,,1680530304583.a15553c96f61b5a8cc6d1615507e7af8.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 13:58:56,032 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 13:58:56,032 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680530336032"}]},"ts":"9223372036854775807"} 2023-04-03 13:58:56,035 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 13:58:56,037 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:56,039 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 58 msec 2023-04-03 13:58:56,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-03 13:58:56,086 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-03 13:58:56,108 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 227), OpenFileDescriptor=520 (was 519) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=307 (was 360), ProcessCount=166 (was 166), AvailableMemoryMB=3580 (was 3514) - AvailableMemoryMB LEAK? - 2023-04-03 13:58:56,117 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=520, MaxFileDescriptor=60000, SystemLoadAverage=307, ProcessCount=166, AvailableMemoryMB=3580 2023-04-03 13:58:56,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:58:56,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:58:56,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:56,122 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 13:58:56,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-03 13:58:56,123 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 13:58:56,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 13:58:56,124 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,124 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 empty. 2023-04-03 13:58:56,125 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,125 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:58:56,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 13:58:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 13:58:56,535 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:58:56,538 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => af4327b273d03b2199ab4cc1b2fa1539, NAME => 'TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp 2023-04-03 13:58:56,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:58:56,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing af4327b273d03b2199ab4cc1b2fa1539, disabling compactions & flushes 2023-04-03 13:58:56,549 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:56,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:56,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. after waiting 0 ms 2023-04-03 13:58:56,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:56,549 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:56,549 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:58:56,553 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 13:58:56,554 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530336553"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530336553"}]},"ts":"1680530336553"} 2023-04-03 13:58:56,554 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 13:58:56,555 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 13:58:56,555 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530336555"}]},"ts":"1680530336555"} 2023-04-03 13:58:56,556 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 13:58:56,558 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, ASSIGN}] 2023-04-03 13:58:56,559 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, ASSIGN 2023-04-03 13:58:56,559 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=false 2023-04-03 13:58:56,711 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=af4327b273d03b2199ab4cc1b2fa1539, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:56,712 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530336711"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530336711"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530336711"}]},"ts":"1680530336711"} 2023-04-03 13:58:56,714 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:58:56,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 13:58:56,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:56,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => af4327b273d03b2199ab4cc1b2fa1539, NAME => 'TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:58:56,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:58:56,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,875 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,877 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,879 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A 2023-04-03 13:58:56,880 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A 2023-04-03 13:58:56,880 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:56,880 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region af4327b273d03b2199ab4cc1b2fa1539 columnFamilyName A 2023-04-03 13:58:56,881 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(310): Store=af4327b273d03b2199ab4cc1b2fa1539/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:56,881 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,882 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B 2023-04-03 13:58:56,882 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B 2023-04-03 13:58:56,882 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:56,882 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region af4327b273d03b2199ab4cc1b2fa1539 columnFamilyName B 2023-04-03 13:58:56,883 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(310): Store=af4327b273d03b2199ab4cc1b2fa1539/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:56,883 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,884 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C 2023-04-03 13:58:56,884 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C 2023-04-03 13:58:56,884 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:56,884 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region af4327b273d03b2199ab4cc1b2fa1539 columnFamilyName C 2023-04-03 13:58:56,885 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(310): Store=af4327b273d03b2199ab4cc1b2fa1539/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:56,885 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:56,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,886 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:58:56,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:56,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 13:58:56,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened af4327b273d03b2199ab4cc1b2fa1539; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64600148, jitterRate=-0.03738278150558472}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:58:56,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:58:56,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., pid=56, masterSystemTime=1680530336867 2023-04-03 13:58:56,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:56,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:56,892 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=af4327b273d03b2199ab4cc1b2fa1539, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:56,892 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530336892"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530336892"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530336892"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530336892"}]},"ts":"1680530336892"} 2023-04-03 13:58:56,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-03 13:58:56,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 in 179 msec 2023-04-03 13:58:56,895 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-03 13:58:56,895 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, ASSIGN in 337 msec 2023-04-03 13:58:56,896 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 13:58:56,896 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530336896"}]},"ts":"1680530336896"} 2023-04-03 13:58:56,896 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 13:58:56,898 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 13:58:56,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 778 msec 2023-04-03 13:58:57,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 13:58:57,228 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-03 13:58:57,228 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x77459ab3 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:57,232 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65a9ccaf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:57,233 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:57,234 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:57,236 DEBUG [Listener at localhost.localdomain/43477] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 13:58:57,238 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 13:58:57,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 13:58:57,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 13:58:57,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 13:58:57,653 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 13:58:57,655 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 13:58:57,655 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 13:58:57,657 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 13:58:57,661 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, REOPEN/MOVE}] 2023-04-03 13:58:57,661 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, REOPEN/MOVE 2023-04-03 13:58:57,662 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=af4327b273d03b2199ab4cc1b2fa1539, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:57,662 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530337662"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530337662"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530337662"}]},"ts":"1680530337662"} 2023-04-03 13:58:57,664 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:58:57,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:57,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing af4327b273d03b2199ab4cc1b2fa1539, disabling compactions & flushes 2023-04-03 13:58:57,818 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:57,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:57,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. after waiting 0 ms 2023-04-03 13:58:57,818 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:57,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 13:58:57,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:57,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:58:57,826 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: af4327b273d03b2199ab4cc1b2fa1539 to self. 2023-04-03 13:58:57,828 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:57,829 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=af4327b273d03b2199ab4cc1b2fa1539, regionState=CLOSED 2023-04-03 13:58:57,829 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530337829"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530337829"}]},"ts":"1680530337829"} 2023-04-03 13:58:57,831 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-03 13:58:57,831 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 in 166 msec 2023-04-03 13:58:57,832 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,39695,1680530174856; forceNewPlan=false, retain=true 2023-04-03 13:58:57,982 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=af4327b273d03b2199ab4cc1b2fa1539, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:57,982 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530337982"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530337982"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530337982"}]},"ts":"1680530337982"} 2023-04-03 13:58:57,984 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:58:58,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:58,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => af4327b273d03b2199ab4cc1b2fa1539, NAME => 'TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.', STARTKEY => '', ENDKEY => ''} 2023-04-03 13:58:58,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 13:58:58,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,146 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,147 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A 2023-04-03 13:58:58,147 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A 2023-04-03 13:58:58,148 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:58,148 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region af4327b273d03b2199ab4cc1b2fa1539 columnFamilyName A 2023-04-03 13:58:58,150 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(310): Store=af4327b273d03b2199ab4cc1b2fa1539/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:58,150 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,151 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B 2023-04-03 13:58:58,151 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B 2023-04-03 13:58:58,151 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:58,152 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region af4327b273d03b2199ab4cc1b2fa1539 columnFamilyName B 2023-04-03 13:58:58,152 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(310): Store=af4327b273d03b2199ab4cc1b2fa1539/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:58,152 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,153 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C 2023-04-03 13:58:58,153 DEBUG [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C 2023-04-03 13:58:58,153 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 13:58:58,154 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region af4327b273d03b2199ab4cc1b2fa1539 columnFamilyName C 2023-04-03 13:58:58,154 INFO [StoreOpener-af4327b273d03b2199ab4cc1b2fa1539-1] regionserver.HStore(310): Store=af4327b273d03b2199ab4cc1b2fa1539/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 13:58:58,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:58,155 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 13:58:58,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened af4327b273d03b2199ab4cc1b2fa1539; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73431604, jitterRate=0.09421616792678833}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 13:58:58,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:58:58,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., pid=61, masterSystemTime=1680530338135 2023-04-03 13:58:58,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:58,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:58,165 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=af4327b273d03b2199ab4cc1b2fa1539, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,165 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530338165"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1680530338165"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680530338165"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680530338165"}]},"ts":"1680530338165"} 2023-04-03 13:58:58,168 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-03 13:58:58,168 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 in 183 msec 2023-04-03 13:58:58,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-03 13:58:58,171 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, REOPEN/MOVE in 507 msec 2023-04-03 13:58:58,173 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-03 13:58:58,173 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-04-03 13:58:58,176 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 934 msec 2023-04-03 13:58:58,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-03 13:58:58,179 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x58f3df8a to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,184 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b84625, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,184 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x0c217430 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,196 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ae8f5b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,196 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5c51ec to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,204 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ddb3e63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,204 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x5be55982 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,212 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@630ea023, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,213 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x2f11e84a to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,221 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec98d63, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,222 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x1ae3ec16 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,234 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8d3d8bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,235 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x7bec0400 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,243 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e6ad437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,243 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x34b8bf14 to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,254 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f6f7187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,255 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x387217bb to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,262 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49c34fbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,263 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(139): Connect 0x7c00a18c to 127.0.0.1:57081 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 13:58:58,271 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5467be78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 13:58:58,278 DEBUG [hconnection-0x40ce9b15-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,278 DEBUG [hconnection-0x2e2fd4d8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,279 DEBUG [hconnection-0x1d4b17ac-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,279 DEBUG [hconnection-0x735fc3a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,281 DEBUG [hconnection-0x20a20dc5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,282 DEBUG [hconnection-0x5b1a5f84-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,282 DEBUG [hconnection-0x647009a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,282 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60762, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,283 DEBUG [hconnection-0x45ac3fa8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,284 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60746, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:58,284 DEBUG [hconnection-0x656e2c78-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,285 DEBUG [hconnection-0x19bc505d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 13:58:58,284 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60764, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,286 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60806, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,286 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60750, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,286 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60792, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,290 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60832, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,290 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60834, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,293 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60778, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,293 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:60822, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 13:58:58,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:58,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:58,299 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:58,299 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:58,299 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:58,299 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:58,300 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,300 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:58,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,301 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:58,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:58,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:58,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:58,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:58,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:58,303 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:58,303 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:58,303 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:58,303 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:58,303 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:58,303 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:58,303 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:58,303 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:58,303 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:58:58,304 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 13:58:58,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,305 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:58:58,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:58:58,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:58,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:58:58,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:58,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:58:58,305 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:58:58,317 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038d8121d45e3b41cab47b0d31ec44ff6e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038d8121d45e3b41cab47b0d31ec44ff6e 2023-04-03 13:58:58,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530398315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530338312, max timestamp=1680530338316Num uniques 859; 2023-04-03 13:58:58,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:58,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680530338312, max timestamp=1680530338316Num uniques 859; 2023-04-03 13:58:58,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:58,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530338312, max timestamp=1680530338316Num uniques 859; 2023-04-03 13:58:58,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:58:58,330 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/6b6a4786de414718bd3dd1b487719b97 2023-04-03 13:58:58,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530398334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530398335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530398336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530398337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530398418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530398438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530398438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530398440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530398441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,623 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530398621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530398640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530398641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530398642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530398643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,749 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/11bb841c96be4eb0863e539c0e55993f 2023-04-03 13:58:58,756 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/0cf805cacdee4219947519395b77c564 2023-04-03 13:58:58,759 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/6b6a4786de414718bd3dd1b487719b97 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/6b6a4786de414718bd3dd1b487719b97 2023-04-03 13:58:58,762 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/6b6a4786de414718bd3dd1b487719b97, entries=150, sequenceid=16, filesize=25.1 K 2023-04-03 13:58:58,763 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/11bb841c96be4eb0863e539c0e55993f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11bb841c96be4eb0863e539c0e55993f 2023-04-03 13:58:58,765 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11bb841c96be4eb0863e539c0e55993f, entries=150, sequenceid=16, filesize=11.6 K 2023-04-03 13:58:58,766 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/0cf805cacdee4219947519395b77c564 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/0cf805cacdee4219947519395b77c564 2023-04-03 13:58:58,768 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/0cf805cacdee4219947519395b77c564, entries=150, sequenceid=16, filesize=11.6 K 2023-04-03 13:58:58,769 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for af4327b273d03b2199ab4cc1b2fa1539 in 465ms, sequenceid=16, compaction requested=false 2023-04-03 13:58:58,769 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:58:58,769 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:58,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:58:58,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:58:58,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:58:58,769 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:58:58,770 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:58,770 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:58,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:58,771 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:58:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:58,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:58:58,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a373410[Count = 0] remaining members to acquire global barrier 2023-04-03 13:58:58,772 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:58:58,772 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:58,772 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:58,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:58,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:58:58,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:58:58,772 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:58:58,772 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,772 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:58:58,773 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:58:58,773 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:58,773 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,773 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:58:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:58,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:58:58,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:58:58,775 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:58:58,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:58:58,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:58:58,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:58:58,775 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:58:58,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:58:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:58:58,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:58:58,777 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:58:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,778 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:58:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:58:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:58,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:58:58,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:58:58,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:58:58,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,780 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:58:58,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:58:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:58:58,781 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:58:58,781 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:58:58,881 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:58:58,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:58:58,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:58:58,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:58:58,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:58:58,883 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:58:58,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:58,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:58:58,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:58:58,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:58,885 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:58:58,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:58:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:58:58,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:58,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:58:58,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:58:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:58:58,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:58:58,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:58:58,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:58:58,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:58:58,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:58:58,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:58:58,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:58:58,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:58:58,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:58:58,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:58:58,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:58:58,886 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-03 13:58:58,886 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:58:58,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:58,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:58:58,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:58,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:58:58,887 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:58:58,900 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304034a23b21895964ff789abc6fac47cd6cb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034a23b21895964ff789abc6fac47cd6cb 2023-04-03 13:58:58,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:58:58,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:58:58,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530398945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530398945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530398947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530398947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:58,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:58,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530398948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:59,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:59,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530399050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:59,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:59,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530399253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:59,305 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/c00c049e7ab542d89b8e3c002a7647a3 2023-04-03 13:58:59,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:59,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530399451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:59,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:59,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530399451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:59,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:59,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530399452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:59,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:59,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530399456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:59,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:58:59,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530399555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:58:59,713 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/b4710114a36141ae9d465cb9b3cd5df6 2023-04-03 13:59:00,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530400057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,121 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/77f69a46fe454d638c9f029cf90fe3a5 2023-04-03 13:59:00,124 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/c00c049e7ab542d89b8e3c002a7647a3 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c00c049e7ab542d89b8e3c002a7647a3 2023-04-03 13:59:00,126 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c00c049e7ab542d89b8e3c002a7647a3, entries=150, sequenceid=50, filesize=25.1 K 2023-04-03 13:59:00,127 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/b4710114a36141ae9d465cb9b3cd5df6 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/b4710114a36141ae9d465cb9b3cd5df6 2023-04-03 13:59:00,129 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/b4710114a36141ae9d465cb9b3cd5df6, entries=150, sequenceid=50, filesize=11.6 K 2023-04-03 13:59:00,130 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/77f69a46fe454d638c9f029cf90fe3a5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/77f69a46fe454d638c9f029cf90fe3a5 2023-04-03 13:59:00,132 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/77f69a46fe454d638c9f029cf90fe3a5, entries=150, sequenceid=50, filesize=11.6 K 2023-04-03 13:59:00,133 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for af4327b273d03b2199ab4cc1b2fa1539 in 1247ms, sequenceid=50, compaction requested=false 2023-04-03 13:59:00,133 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:00,133 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:00,133 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:00,133 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:00,133 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,133 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:00,133 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:00,134 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,134 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:00,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:00,134 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,134 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:00,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:00,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72c1c41e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:00,136 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:00,136 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,136 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,136 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:00,136 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:00,136 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:00,136 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,137 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:00,137 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:00,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:00,137 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:00,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:00,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:00,139 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:00,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:00,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:00,139 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:00,139 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:00,140 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,140 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:00,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,140 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,140 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:00,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,141 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:00,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:00,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,144 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:00,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:00,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:00,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:00,144 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,145 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:00,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:00,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:00,145 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:00,145 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:00,245 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:00,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:00,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:00,250 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:00,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:00,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:00,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:00,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,250 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,251 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:00,251 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:00,251 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,251 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:00,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:00,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:00,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:00,252 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:00,252 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:00,252 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:00,252 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:00,252 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:00,252 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:00,252 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:00,252 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 13:59:00,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:00,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:00,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:00,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:00,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:00,253 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:00,274 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038e0a5da400d5412fa340a8820aa186ef to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038e0a5da400d5412fa340a8820aa186ef 2023-04-03 13:59:00,279 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/59c74cce6ac54bce9177c25f9644e774 2023-04-03 13:59:00,288 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/016b4ae1037d4035be0b568899ae65a8 2023-04-03 13:59:00,316 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/7de0821458f34b049737f92aabc9ae45 2023-04-03 13:59:00,321 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/59c74cce6ac54bce9177c25f9644e774 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/59c74cce6ac54bce9177c25f9644e774 2023-04-03 13:59:00,324 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/59c74cce6ac54bce9177c25f9644e774, entries=150, sequenceid=62, filesize=25.1 K 2023-04-03 13:59:00,325 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/016b4ae1037d4035be0b568899ae65a8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/016b4ae1037d4035be0b568899ae65a8 2023-04-03 13:59:00,328 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/016b4ae1037d4035be0b568899ae65a8, entries=150, sequenceid=62, filesize=11.6 K 2023-04-03 13:59:00,328 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/7de0821458f34b049737f92aabc9ae45 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7de0821458f34b049737f92aabc9ae45 2023-04-03 13:59:00,331 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7de0821458f34b049737f92aabc9ae45, entries=150, sequenceid=62, filesize=11.6 K 2023-04-03 13:59:00,332 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for af4327b273d03b2199ab4cc1b2fa1539 in 80ms, sequenceid=62, compaction requested=true 2023-04-03 13:59:00,332 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:00,332 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:00,332 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:00,332 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:00,332 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,332 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:00,332 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:00,333 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:00,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:00,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:00,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:00,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:00,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a6fe5a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:00,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:00,335 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,335 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:00,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:00,336 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:00,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:00,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:00,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:00,337 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:00,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:00,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:00,339 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:00,339 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:00,339 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:00,339 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:00,339 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:00,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:00,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:00,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,341 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:00,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,342 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:00,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:00,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:00,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:00,344 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:00,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:00,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:00,345 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:00,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:00,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:00,345 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:00,445 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:00,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:00,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:00,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:00,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:00,462 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:00,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:00,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:00,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:00,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,463 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:00,463 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:00,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:00,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:00,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:00,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:00,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:00,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:00,474 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:00,474 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:00,474 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:00,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:00,482 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:00,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:00,482 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 13:59:00,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:00,482 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:00,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:00,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:00,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:00,483 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:00,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:00,484 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:00,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530340486, max timestamp=1680530340508Num uniques 859; 2023-04-03 13:59:00,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:00,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530340486, max timestamp=1680530340508Num uniques 859; 2023-04-03 13:59:00,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530340486, max timestamp=1680530340508Num uniques 859; 2023-04-03 13:59:00,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:00,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:00,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680530340521, max timestamp=1680530340528Num uniques 1018; 2023-04-03 13:59:00,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530400527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-03 13:59:00,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680530340521, max timestamp=1680530340528Num uniques 1018; 2023-04-03 13:59:00,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-03 13:59:00,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1680530340521, max timestamp=1680530340528Num uniques 1018; 2023-04-03 13:59:00,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-03 13:59:00,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530400554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530400554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530400555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530400632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530400659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530400660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530400660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530400837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530400862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530400863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:00,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530400864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:00,919 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304030c41e4201842489abd87db95cec71d07 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030c41e4201842489abd87db95cec71d07 2023-04-03 13:59:00,949 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=71, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/92d1f2315f1543c88cdee95d51bae49f 2023-04-03 13:59:00,959 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=71 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/86dc8ae0491d4a3c9dff75432c4f9762 2023-04-03 13:59:01,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530401062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,085 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 13:59:01,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530401140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530401166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530401167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530401167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,368 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=71 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/6e05c900670e4cbaa0de163a36a38a0b 2023-04-03 13:59:01,371 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/92d1f2315f1543c88cdee95d51bae49f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/92d1f2315f1543c88cdee95d51bae49f 2023-04-03 13:59:01,374 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/92d1f2315f1543c88cdee95d51bae49f, entries=300, sequenceid=71, filesize=45.5 K 2023-04-03 13:59:01,374 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/86dc8ae0491d4a3c9dff75432c4f9762 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/86dc8ae0491d4a3c9dff75432c4f9762 2023-04-03 13:59:01,377 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/86dc8ae0491d4a3c9dff75432c4f9762, entries=100, sequenceid=71, filesize=9.3 K 2023-04-03 13:59:01,378 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/6e05c900670e4cbaa0de163a36a38a0b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/6e05c900670e4cbaa0de163a36a38a0b 2023-04-03 13:59:01,381 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/6e05c900670e4cbaa0de163a36a38a0b, entries=100, sequenceid=71, filesize=9.3 K 2023-04-03 13:59:01,381 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for af4327b273d03b2199ab4cc1b2fa1539 in 899ms, sequenceid=71, compaction requested=true 2023-04-03 13:59:01,382 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:01,382 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:01,382 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:01,382 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:01,382 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,382 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:01,382 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:01,382 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,383 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:01,383 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:01,383 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:01,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:01,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f8dfd18[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:01,384 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:01,384 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:01,384 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:01,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:01,385 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,385 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:01,385 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:01,385 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:01,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:01,387 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:01,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:01,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:01,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:01,387 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:01,388 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:01,388 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,389 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:01,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:01,389 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:01,390 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:01,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:01,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:01,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:01,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:01,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:01,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:01,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:01,392 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:01,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:01,392 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:01,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:01,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:01,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,492 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:01,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:01,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:01,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:01,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:01,496 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:01,496 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:01,496 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:01,496 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:01,497 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,497 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:01,497 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,497 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:01,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:01,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:01,498 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,498 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:01,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:01,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:01,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:01,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:01,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:01,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:01,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:01,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:01,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:01,498 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:01,498 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:01,499 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:01,499 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=147.60 KB heapSize=409.92 KB 2023-04-03 13:59:01,499 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:01,499 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:01,499 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:01,499 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:01,499 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:01,499 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:01,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:01,645 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:01,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530401659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530401671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530401672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530401672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530401763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:01,916 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040314de667e1df8407688285f65a45d4902 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040314de667e1df8407688285f65a45d4902 2023-04-03 13:59:01,920 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=124, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/aa9db53527b14c588d1f9e296d8ae2c9 2023-04-03 13:59:01,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:01,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530401966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:02,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530402270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,327 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/bc16d9d7a24843a28efc9b89ec06d976 2023-04-03 13:59:02,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:02,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530402680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:02,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530402682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:02,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530402682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,736 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/9702d717df714b138099e95f653cf046 2023-04-03 13:59:02,739 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/aa9db53527b14c588d1f9e296d8ae2c9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa9db53527b14c588d1f9e296d8ae2c9 2023-04-03 13:59:02,741 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa9db53527b14c588d1f9e296d8ae2c9, entries=150, sequenceid=124, filesize=25.1 K 2023-04-03 13:59:02,742 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/bc16d9d7a24843a28efc9b89ec06d976 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bc16d9d7a24843a28efc9b89ec06d976 2023-04-03 13:59:02,744 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bc16d9d7a24843a28efc9b89ec06d976, entries=150, sequenceid=124, filesize=11.6 K 2023-04-03 13:59:02,745 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/9702d717df714b138099e95f653cf046 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/9702d717df714b138099e95f653cf046 2023-04-03 13:59:02,747 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/9702d717df714b138099e95f653cf046, entries=150, sequenceid=124, filesize=11.6 K 2023-04-03 13:59:02,748 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~409.88 KB/419712, currentSize=40.25 KB/41220 for af4327b273d03b2199ab4cc1b2fa1539 in 1249ms, sequenceid=124, compaction requested=true 2023-04-03 13:59:02,748 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:02,748 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:02,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:02,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:02,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:02,748 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:02,749 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,749 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,749 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,749 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:02,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c69c54d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:02,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:02,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:02,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:02,751 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:02,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:02,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,752 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:02,752 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:02,752 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:02,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:02,754 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:02,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:02,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:02,754 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:02,754 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:02,754 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,754 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,754 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:02,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:02,755 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:02,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:02,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:02,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:02,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:02,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:02,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:02,758 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,758 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,758 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,758 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,758 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:02,758 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:02,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:02,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:02,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:02,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:59:02,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:02,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:02,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:02,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:02,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:02,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:02,793 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304037618c126d67d425e903bb0f10fc26a76 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037618c126d67d425e903bb0f10fc26a76 2023-04-03 13:59:02,803 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/c0f91c0312f04ae6a002a0a468f2c151 2023-04-03 13:59:02,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/2e0ef5f1f0ef4f6f9dec43316f2718bf 2023-04-03 13:59:02,858 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:02,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:02,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530342842, max timestamp=1680530342857Num uniques 859; 2023-04-03 13:59:02,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530342842, max timestamp=1680530342857Num uniques 859; 2023-04-03 13:59:02,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530342842, max timestamp=1680530342857Num uniques 859; 2023-04-03 13:59:02,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:02,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:02,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:02,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:02,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:02,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:02,861 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:02,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:02,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:02,861 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:02,862 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,862 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,862 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:02,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:02,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,863 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:02,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:02,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:02,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:02,864 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:02,864 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:02,864 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:02,864 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:02,864 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:02,865 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:02,866 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,866 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,866 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,866 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:02,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50f405ea[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:02,867 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:02,867 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,868 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,868 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:02,868 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:02,868 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:02,869 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,869 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:02,869 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:02,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:02,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:02,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:02,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:02,871 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:02,871 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:02,871 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:02,871 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:02,871 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:02,872 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,872 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,872 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:02,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,873 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,874 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,875 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:02,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:02,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:02,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:02,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:02,875 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:02,875 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:02,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:02,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:02,875 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,876 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:02,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530402931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,975 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:02,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:02,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:02,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:02,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:02,978 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:02,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:02,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:02,978 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:02,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,979 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:02,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:02,979 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,980 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:02,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:02,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:02,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:02,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:02,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:02,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:02,980 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:02,980 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:02,981 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:02,981 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:02,981 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:02,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:02,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:02,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:02,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:02,981 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,981 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,982 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:02,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14646580[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:02,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:02,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,983 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,983 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:02,983 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:02,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,983 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:02,983 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:02,984 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:02,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,984 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:02,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:02,986 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:02,986 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:02,986 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:02,986 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:02,986 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:02,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:02,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:02,987 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:02,988 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,989 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,990 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:02,990 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:02,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:02,990 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,990 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:02,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:02,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,991 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:02,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:02,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:02,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:02,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:02,991 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:02,991 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:02,991 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:03,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530403038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:03,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530403069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,072 DEBUG [Thread-3437] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4124 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:59:03,091 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:03,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:03,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:03,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:03,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:03,094 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:03,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:03,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:03,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,094 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,095 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,095 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:03,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:03,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:03,096 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,096 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:03,096 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:03,096 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:03,096 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:03,097 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,097 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,097 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,097 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:03,099 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:03,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24b13f29[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:03,099 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,099 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:03,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:03,099 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,099 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:03,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:03,100 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,100 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,100 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,100 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:03,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:03,102 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:03,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:03,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:03,102 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:03,102 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:03,103 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,103 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,103 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,104 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,105 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,106 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:03,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:03,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,107 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,107 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:03,107 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,107 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,207 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:03,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:03,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:03,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:03,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:03,210 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:03,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:03,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:03,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,210 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,211 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,211 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,211 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,211 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:03,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:03,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:03,212 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,212 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:03,212 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:03,212 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:03,212 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:03,213 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,213 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,213 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,213 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:03,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4610c55f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:03,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:03,214 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,215 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,215 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:03,215 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:03,215 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:03,215 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,215 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:03,217 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,217 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:03,217 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,217 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,218 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/308492eba68f4681ad510b69c6f0e40b 2023-04-03 13:59:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:03,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:03,219 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:03,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:03,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:03,219 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:03,219 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:03,220 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,220 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,220 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,221 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/c0f91c0312f04ae6a002a0a468f2c151 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c0f91c0312f04ae6a002a0a468f2c151 2023-04-03 13:59:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,222 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:03,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,224 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:03,224 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,224 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,224 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c0f91c0312f04ae6a002a0a468f2c151, entries=200, sequenceid=134, filesize=32.1 K 2023-04-03 13:59:03,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/2e0ef5f1f0ef4f6f9dec43316f2718bf as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2e0ef5f1f0ef4f6f9dec43316f2718bf 2023-04-03 13:59:03,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2e0ef5f1f0ef4f6f9dec43316f2718bf, entries=150, sequenceid=134, filesize=11.8 K 2023-04-03 13:59:03,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/308492eba68f4681ad510b69c6f0e40b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/308492eba68f4681ad510b69c6f0e40b 2023-04-03 13:59:03,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/308492eba68f4681ad510b69c6f0e40b, entries=150, sequenceid=134, filesize=11.8 K 2023-04-03 13:59:03,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for af4327b273d03b2199ab4cc1b2fa1539 in 457ms, sequenceid=134, compaction requested=true 2023-04-03 13:59:03,231 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:03,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:03,231 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:59:03,231 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:59:03,231 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:03,232 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:03,233 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 182386 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:59:03,233 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69122 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:59:03,233 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/A is initiating minor compaction (all files) 2023-04-03 13:59:03,233 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/B is initiating minor compaction (all files) 2023-04-03 13:59:03,233 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/A in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,233 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/B in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,233 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/6b6a4786de414718bd3dd1b487719b97, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c00c049e7ab542d89b8e3c002a7647a3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/59c74cce6ac54bce9177c25f9644e774, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/92d1f2315f1543c88cdee95d51bae49f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa9db53527b14c588d1f9e296d8ae2c9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c0f91c0312f04ae6a002a0a468f2c151] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=178.1 K 2023-04-03 13:59:03,233 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11bb841c96be4eb0863e539c0e55993f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/b4710114a36141ae9d465cb9b3cd5df6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/016b4ae1037d4035be0b568899ae65a8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/86dc8ae0491d4a3c9dff75432c4f9762, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bc16d9d7a24843a28efc9b89ec06d976, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2e0ef5f1f0ef4f6f9dec43316f2718bf] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=67.5 K 2023-04-03 13:59:03,234 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 6b6a4786de414718bd3dd1b487719b97, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680530338300 2023-04-03 13:59:03,234 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 11bb841c96be4eb0863e539c0e55993f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680530338300 2023-04-03 13:59:03,234 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting c00c049e7ab542d89b8e3c002a7647a3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680530338334 2023-04-03 13:59:03,234 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting b4710114a36141ae9d465cb9b3cd5df6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680530338334 2023-04-03 13:59:03,234 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 59c74cce6ac54bce9177c25f9644e774, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680530338929 2023-04-03 13:59:03,234 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 016b4ae1037d4035be0b568899ae65a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680530338929 2023-04-03 13:59:03,234 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 92d1f2315f1543c88cdee95d51bae49f, keycount=300, bloomtype=ROW, size=45.5 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1680530340471 2023-04-03 13:59:03,234 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 86dc8ae0491d4a3c9dff75432c4f9762, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1680530340474 2023-04-03 13:59:03,235 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting aa9db53527b14c588d1f9e296d8ae2c9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680530340552 2023-04-03 13:59:03,235 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bc16d9d7a24843a28efc9b89ec06d976, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680530340552 2023-04-03 13:59:03,235 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting c0f91c0312f04ae6a002a0a468f2c151, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530341647 2023-04-03 13:59:03,235 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2e0ef5f1f0ef4f6f9dec43316f2718bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530341653 2023-04-03 13:59:03,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:03,243 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-03 13:59:03,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:03,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:03,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:03,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:03,243 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:03,243 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:03,243 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#B#compaction#460 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:03,256 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#A#compaction#459 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:03,257 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/d99472743cd04bd4aad954388ceb2640 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d99472743cd04bd4aad954388ceb2640 2023-04-03 13:59:03,262 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/B of af4327b273d03b2199ab4cc1b2fa1539 into d99472743cd04bd4aad954388ceb2640(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:03,262 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:03,262 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/B, priority=10, startTime=1680530343231; duration=0sec 2023-04-03 13:59:03,262 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:03,262 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:59:03,263 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69122 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:59:03,263 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/C is initiating minor compaction (all files) 2023-04-03 13:59:03,264 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/C in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,264 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/0cf805cacdee4219947519395b77c564, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/77f69a46fe454d638c9f029cf90fe3a5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7de0821458f34b049737f92aabc9ae45, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/6e05c900670e4cbaa0de163a36a38a0b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/9702d717df714b138099e95f653cf046, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/308492eba68f4681ad510b69c6f0e40b] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=67.5 K 2023-04-03 13:59:03,264 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 0cf805cacdee4219947519395b77c564, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680530338300 2023-04-03 13:59:03,264 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 77f69a46fe454d638c9f029cf90fe3a5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680530338334 2023-04-03 13:59:03,264 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 7de0821458f34b049737f92aabc9ae45, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680530338929 2023-04-03 13:59:03,265 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 6e05c900670e4cbaa0de163a36a38a0b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1680530340474 2023-04-03 13:59:03,265 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 9702d717df714b138099e95f653cf046, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1680530340552 2023-04-03 13:59:03,265 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 308492eba68f4681ad510b69c6f0e40b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530341653 2023-04-03 13:59:03,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:03,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530403267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,273 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#C#compaction#462 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:03,283 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/86fe47bd118448c89e67bc6f29939eb1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/86fe47bd118448c89e67bc6f29939eb1 2023-04-03 13:59:03,286 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/C of af4327b273d03b2199ab4cc1b2fa1539 into 86fe47bd118448c89e67bc6f29939eb1(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:03,286 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:03,286 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/C, priority=10, startTime=1680530343232; duration=0sec 2023-04-03 13:59:03,286 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:03,324 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:03,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:03,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:03,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:03,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:03,327 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:03,327 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,327 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:03,327 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:03,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,328 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,329 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,329 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:03,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:03,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:03,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:03,329 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:03,330 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,330 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:03,330 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:03,330 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:03,330 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:03,331 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,331 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,331 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,331 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:03,332 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:03,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bb4322[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:03,332 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,333 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:03,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:03,333 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,333 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:03,333 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:03,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:03,334 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,334 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:03,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:03,336 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:03,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:03,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:03,336 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:03,336 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:03,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,338 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,339 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,339 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,339 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:03,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:03,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:03,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,340 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,340 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,340 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:03,340 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:03,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530403371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,440 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:03,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:03,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:03,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:03,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:03,443 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:03,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:03,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:03,443 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,444 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,444 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:03,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:03,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:03,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:03,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,445 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:03,450 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:03,450 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:03,450 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:03,450 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:03,450 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:03,458 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,458 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,459 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:03,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c46810[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:03,460 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:03,460 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,461 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:03,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:03,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:03,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,462 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:03,462 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:03,464 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:03,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:03,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:03,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:03,464 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:03,464 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:03,465 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,465 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,465 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,465 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,466 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:03,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:03,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:03,468 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:03,468 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,468 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,468 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,568 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:03,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:03,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:03,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:03,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:03,571 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:03,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:03,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:03,571 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,572 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,572 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:03,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:03,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:03,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:03,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:03,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:03,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:03,573 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:03,573 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,573 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:03,574 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:03,574 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:03,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:03,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:03,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:03,574 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,574 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,575 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:03,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca15776[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:03,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:03,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,576 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,576 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:03,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:03,576 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:03,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:03,577 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530403575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:03,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:03,579 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:03,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:03,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:03,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:03,579 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:03,581 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,581 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,581 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,581 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:03,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:03,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:03,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,584 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,584 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,583 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,584 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:03,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,584 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,664 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040338dcdcf535ff4197b69f89c07262a61d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040338dcdcf535ff4197b69f89c07262a61d 2023-04-03 13:59:03,667 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=172, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/d1299f452cbe4571852af221ddb5c689 2023-04-03 13:59:03,684 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:03,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:03,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:03,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:03,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:03,687 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:03,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:03,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:03,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,687 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,688 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,688 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:03,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:03,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:03,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:03,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:03,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:03,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:03,689 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:03,689 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:03,690 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,690 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:03,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:03,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c5b5580[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:03,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,692 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,692 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:03,692 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:03,692 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:03,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,692 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:03,693 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:03,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:03,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:03,694 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:03,694 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:03,694 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:03,695 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:03,695 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:03,695 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,695 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,695 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,696 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,697 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,698 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,698 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,698 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,698 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:03,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:03,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,698 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,698 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:03,698 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,698 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,698 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:03,699 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,699 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,699 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,798 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:03,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:03,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:03,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:03,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:03,801 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:03,801 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,801 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:03,801 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:03,801 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,801 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,802 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,802 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,802 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,802 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:03,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:03,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:03,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:03,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:03,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:03,803 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:03,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:03,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:03,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:03,804 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:03,804 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:03,805 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,805 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:03,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b97a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:03,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:03,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,807 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,807 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:03,807 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:03,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:03,807 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:03,808 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:03,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:03,810 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:03,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:03,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:03,810 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:03,810 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:03,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,811 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,812 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,813 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,815 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,815 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,815 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:03,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:03,815 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:03,815 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,816 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,816 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:03,816 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,816 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,816 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,816 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,816 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530403881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,916 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:03,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:03,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:03,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:03,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:03,918 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:03,919 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,919 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:03,919 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:03,919 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,919 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,920 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,920 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,920 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,920 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:03,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:03,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:03,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:03,921 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,921 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:03,921 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:03,921 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:03,922 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,922 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,923 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,923 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:03,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:03,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22b3d580[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:03,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:03,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:03,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:03,924 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:03,925 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:03,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:03,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:03,929 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:03,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:03,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:03,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:03,929 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:03,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,933 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:03,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:03,934 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:03,935 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:03,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:03,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:03,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:03,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:03,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:03,937 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:03,937 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,937 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:03,937 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,037 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:04,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:04,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:04,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:04,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:04,040 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:04,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:04,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:04,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,041 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,041 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,041 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:04,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:04,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:04,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:04,042 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:04,042 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:04,042 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:04,042 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:04,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:04,043 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,043 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,043 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,043 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:04,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:04,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48f2c47[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:04,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,045 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:04,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:04,045 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,045 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:04,045 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:04,046 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,046 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,046 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,046 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:04,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:04,049 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:04,049 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:04,049 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:04,049 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:04,049 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:04,050 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,050 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,050 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,051 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,053 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:04,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,053 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,053 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:04,053 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/293a2e65aafb4e768563c52a55a50a94 2023-04-03 13:59:04,153 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:04,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:04,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:04,156 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:04,156 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,156 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:04,156 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:04,156 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,157 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,157 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,157 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,157 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,157 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:04,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:04,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:04,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:04,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,158 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:04,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:04,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:04,159 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,159 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:04,159 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:04,159 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:04,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:04,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:04,159 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:04,160 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,160 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:04,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@324e8e03[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:04,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:04,161 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,161 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,162 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:04,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:04,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:04,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:04,162 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,162 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:04,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:04,164 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:04,164 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:04,164 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:04,164 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:04,164 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:04,165 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,165 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,165 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,165 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,166 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:04,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:04,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,167 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:04,167 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,167 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,167 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:04,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:04,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,168 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,268 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:04,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:04,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:04,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:04,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:04,270 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:04,270 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,270 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:04,270 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:04,271 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,271 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,271 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,271 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,271 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,271 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:04,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:04,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:04,272 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:04,272 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,272 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:04,272 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:04,272 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:04,272 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,273 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:04,273 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:04,273 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,273 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:04,273 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:04,273 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,273 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:04,273 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:04,274 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,274 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,274 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,274 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:04,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f3d3f59[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:04,275 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:04,275 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,276 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,276 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:04,276 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:04,276 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:04,276 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,276 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:04,276 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,276 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,277 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:04,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:04,278 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:04,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:04,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:04,278 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:04,278 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:04,279 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,280 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,280 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,280 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,282 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,282 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,282 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:04,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:04,282 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,282 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,283 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:04,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:04,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,283 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,283 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:04,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,383 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:04,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:04,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:04,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:04,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:04,386 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:04,386 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,386 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:04,386 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:04,386 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,386 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:04,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530404386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,387 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,387 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:04,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:04,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:04,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:04,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:04,388 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,388 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:04,388 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:04,388 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:04,388 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:04,389 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,389 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:04,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:04,390 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:04,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@239ee6eb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:04,390 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,391 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,391 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:04,391 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:04,391 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:04,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,391 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:04,392 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:04,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,392 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:04,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:04,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:04,394 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:04,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:04,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:04,394 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:04,394 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:04,395 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,395 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,395 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,395 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,396 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,397 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:04,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:04,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,398 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:04,398 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:04,397 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,398 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,481 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/93b854ec6e674fdd8400f993de3318b1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/93b854ec6e674fdd8400f993de3318b1 2023-04-03 13:59:04,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/7d45d23797b445ef9f2ffff5932d8006 2023-04-03 13:59:04,486 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/A of af4327b273d03b2199ab4cc1b2fa1539 into 93b854ec6e674fdd8400f993de3318b1(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-03 13:59:04,486 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:04,486 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/A, priority=10, startTime=1680530343231; duration=1sec 2023-04-03 13:59:04,486 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:04,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/d1299f452cbe4571852af221ddb5c689 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d1299f452cbe4571852af221ddb5c689 2023-04-03 13:59:04,498 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:04,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d1299f452cbe4571852af221ddb5c689, entries=150, sequenceid=172, filesize=25.3 K 2023-04-03 13:59:04,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:04,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/293a2e65aafb4e768563c52a55a50a94 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/293a2e65aafb4e768563c52a55a50a94 2023-04-03 13:59:04,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:04,501 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:04,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:04,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:04,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,501 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,502 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,502 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,503 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,503 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/293a2e65aafb4e768563c52a55a50a94, entries=150, sequenceid=172, filesize=11.8 K 2023-04-03 13:59:04,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:04,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/7d45d23797b445ef9f2ffff5932d8006 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7d45d23797b445ef9f2ffff5932d8006 2023-04-03 13:59:04,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:04,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:04,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:04,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:04,504 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:04,504 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:04,505 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,505 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:04,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7d45d23797b445ef9f2ffff5932d8006, entries=150, sequenceid=172, filesize=11.8 K 2023-04-03 13:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:04,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0e1bcb[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:04,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:04,507 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,507 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for af4327b273d03b2199ab4cc1b2fa1539 in 1264ms, sequenceid=172, compaction requested=false 2023-04-03 13:59:04,507 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:04,508 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,508 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:04,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:04,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:04,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:04,508 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,508 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,509 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:04,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:04,511 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:04,511 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:04,511 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:04,511 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:04,511 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:04,511 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:04,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,512 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:04,512 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,513 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:04,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,515 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:04,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:04,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,515 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,515 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:04,515 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:04,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:04,615 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:04,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:04,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:04,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:04,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:04,618 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:04,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:04,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:04,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,618 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,619 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,619 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:04,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:04,619 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,619 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:04,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:04,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:04,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:04,622 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:04,622 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:04,622 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:04,622 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:04,622 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:04,622 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:04,622 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:04,622 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 13:59:04,622 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:04,622 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:04,622 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:04,622 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:04,622 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:04,622 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:04,637 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040385df8f1ca17a42b8a55fefc035a6a6e8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040385df8f1ca17a42b8a55fefc035a6a6e8 2023-04-03 13:59:04,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:04,685 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:04,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530344706, max timestamp=1680530344710Num uniques 859; 2023-04-03 13:59:04,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530344706, max timestamp=1680530344710Num uniques 859; 2023-04-03 13:59:04,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:59:04,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:59:04,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680530344706, max timestamp=1680530344710Num uniques 859; 2023-04-03 13:59:04,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:59:04,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:04,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530404729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:04,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530404730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:04,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530404733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:04,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530404834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:04,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530404834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:04,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:04,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530404836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530405035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530405036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530405038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,041 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=184, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/aa23d20d85434d2eac5d028b314a64c0 2023-04-03 13:59:05,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530405339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530405339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530405342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530405389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,450 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/11ab2c30f3a546c096344d239b900c2f 2023-04-03 13:59:05,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530405842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530405843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:05,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530405846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,865 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/d37c77462b494696a8a97cf08d95073a 2023-04-03 13:59:05,868 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/aa23d20d85434d2eac5d028b314a64c0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa23d20d85434d2eac5d028b314a64c0 2023-04-03 13:59:05,871 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa23d20d85434d2eac5d028b314a64c0, entries=150, sequenceid=184, filesize=25.3 K 2023-04-03 13:59:05,871 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/11ab2c30f3a546c096344d239b900c2f as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11ab2c30f3a546c096344d239b900c2f 2023-04-03 13:59:05,874 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11ab2c30f3a546c096344d239b900c2f, entries=150, sequenceid=184, filesize=11.8 K 2023-04-03 13:59:05,875 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/d37c77462b494696a8a97cf08d95073a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d37c77462b494696a8a97cf08d95073a 2023-04-03 13:59:05,877 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d37c77462b494696a8a97cf08d95073a, entries=150, sequenceid=184, filesize=11.8 K 2023-04-03 13:59:05,877 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for af4327b273d03b2199ab4cc1b2fa1539 in 1255ms, sequenceid=184, compaction requested=true 2023-04-03 13:59:05,878 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:05,878 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:05,878 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:05,878 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:05,878 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,878 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:05,878 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:05,879 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:05,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:05,879 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:05,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:05,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:05,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1876b89f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:05,880 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:05,880 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:05,880 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:05,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:05,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,881 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:05,881 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:05,881 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:05,881 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:05,881 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:05,881 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,881 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:05,882 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:05,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:05,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:05,883 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:05,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:05,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:05,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:05,884 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:05,884 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,884 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:05,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,884 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:05,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:05,885 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:05,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:05,886 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:05,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,887 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,887 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:05,887 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:05,887 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:05,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:05,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:05,888 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:05,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:05,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:05,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:05,887 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:05,888 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:05,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:05,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:05,888 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:05,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:05,988 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:05,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:05,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:05,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:05,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:05,991 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:05,991 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:05,991 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:05,991 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:05,991 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,991 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:05,993 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,993 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:05,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:05,993 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:05,993 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:05,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:05,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:05,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:05,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:05,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:05,994 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:05,994 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:05,994 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:05,994 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:05,994 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:05,994 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:05,994 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:05,994 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-03 13:59:05,994 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:05,994 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:05,994 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:05,995 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:05,995 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:05,995 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:06,404 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038df1eaaad24d43568c6015d196d8b383 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038df1eaaad24d43568c6015d196d8b383 2023-04-03 13:59:06,808 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=221, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/ca2a3a24f68a4116a7e1489be6c190cb 2023-04-03 13:59:06,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:06,854 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:06,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:06,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530406858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:06,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:06,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530406859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:06,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:06,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530406861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:06,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:06,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530406962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:06,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:06,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530406962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:06,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:06,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530406964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:07,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530407082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,085 DEBUG [Thread-3437] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:59:07,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:07,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530407165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:07,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530407165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:07,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530407167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,216 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/003f26533cd74e699a91b07f2417e10c 2023-04-03 13:59:07,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:07,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60778 deadline: 1680530407403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,405 DEBUG [Thread-3435] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:59:07,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:07,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530407468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:07,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530407470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:07,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530407471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,624 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=221 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/2770ca780f114f26837155a7b4099964 2023-04-03 13:59:07,627 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/ca2a3a24f68a4116a7e1489be6c190cb as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/ca2a3a24f68a4116a7e1489be6c190cb 2023-04-03 13:59:07,630 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/ca2a3a24f68a4116a7e1489be6c190cb, entries=150, sequenceid=221, filesize=25.3 K 2023-04-03 13:59:07,630 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/003f26533cd74e699a91b07f2417e10c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/003f26533cd74e699a91b07f2417e10c 2023-04-03 13:59:07,633 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/003f26533cd74e699a91b07f2417e10c, entries=150, sequenceid=221, filesize=11.8 K 2023-04-03 13:59:07,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/2770ca780f114f26837155a7b4099964 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2770ca780f114f26837155a7b4099964 2023-04-03 13:59:07,636 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2770ca780f114f26837155a7b4099964, entries=150, sequenceid=221, filesize=11.8 K 2023-04-03 13:59:07,636 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for af4327b273d03b2199ab4cc1b2fa1539 in 1642ms, sequenceid=221, compaction requested=true 2023-04-03 13:59:07,636 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:07,636 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:07,636 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:07,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:07,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:07,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:07,637 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:07,637 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:07,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:07,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:07,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:07,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:07,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec78b3f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:07,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:07,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:07,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:07,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:07,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:07,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:07,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:07,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:07,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:07,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:07,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:07,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:07,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:07,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:07,642 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:07,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:07,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:07,642 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:07,643 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:07,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:07,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:07,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:07,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:07,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:07,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:07,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:07,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:07,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:07,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:07,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:07,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:07,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:07,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:07,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:07,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:07,647 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:07,646 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:07,646 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:07,647 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:07,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:07,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:07,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,747 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:07,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:07,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:07,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:07,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:07,750 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:07,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:07,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:07,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:07,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,750 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:07,751 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:07,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:07,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:07,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:07,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:07,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:07,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:07,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:07,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:07,752 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:07,752 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:07,752 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:07,752 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:07,752 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:07,752 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:07,752 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:07,752 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 13:59:07,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:07,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:07,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:07,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:07,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:07,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:07,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:07,976 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:08,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1680530347987, max timestamp=1680530347998Num uniques 859; 2023-04-03 13:59:08,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1680530347987, max timestamp=1680530347998Num uniques 859; 2023-04-03 13:59:08,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:59:08,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 13:59:08,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7591, totalHeapSize=17504, min timestamp=1680530347987, max timestamp=1680530347998Num uniques 859; 2023-04-03 13:59:08,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39329, new segment data size=7591, suffix heap size=117064,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=166 2023-04-03 13:59:08,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530408040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530408040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530408044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530408144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530408144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530408148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,161 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403061e0ba05ee446f095c41ce5e2c50e42 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403061e0ba05ee446f095c41ce5e2c50e42 2023-04-03 13:59:08,166 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/dc9de894e9ea45f4a7c29c4ad08c7ab8 2023-04-03 13:59:08,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530408348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530408348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530408351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,573 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/52827c2b03fb49b4bba3fc4c23ade9ad 2023-04-03 13:59:08,580 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/d2d48e29fceb45908cb20bbb211d31f0 2023-04-03 13:59:08,583 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/dc9de894e9ea45f4a7c29c4ad08c7ab8 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/dc9de894e9ea45f4a7c29c4ad08c7ab8 2023-04-03 13:59:08,585 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/dc9de894e9ea45f4a7c29c4ad08c7ab8, entries=150, sequenceid=230, filesize=25.3 K 2023-04-03 13:59:08,586 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/52827c2b03fb49b4bba3fc4c23ade9ad as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/52827c2b03fb49b4bba3fc4c23ade9ad 2023-04-03 13:59:08,589 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/52827c2b03fb49b4bba3fc4c23ade9ad, entries=150, sequenceid=230, filesize=11.8 K 2023-04-03 13:59:08,589 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/d2d48e29fceb45908cb20bbb211d31f0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d2d48e29fceb45908cb20bbb211d31f0 2023-04-03 13:59:08,592 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d2d48e29fceb45908cb20bbb211d31f0, entries=150, sequenceid=230, filesize=11.8 K 2023-04-03 13:59:08,592 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=138.97 KB/142302 for af4327b273d03b2199ab4cc1b2fa1539 in 840ms, sequenceid=230, compaction requested=true 2023-04-03 13:59:08,592 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:08,592 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:08,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:08,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:08,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:08,592 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:08,593 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,593 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:08,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aba91b2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:08,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:08,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,595 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:08,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:08,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,595 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:08,595 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:08,596 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:08,596 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:08,596 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,596 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:08,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:08,598 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:08,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:08,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:08,598 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:08,598 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:08,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,598 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,599 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,599 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,600 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:08,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:08,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:08,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:08,601 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:08,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:08,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:08,601 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,602 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:08,602 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:08,602 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:08,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:08,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:08,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:08,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=145.68 KB heapSize=416.48 KB 2023-04-03 13:59:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:08,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:08,660 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530408658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530408659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530408660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,664 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038649727dbe43485a9d4d0a695d6227cc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038649727dbe43485a9d4d0a695d6227cc 2023-04-03 13:59:08,667 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/339ca567b9b44b9ca86516570f61874a 2023-04-03 13:59:08,702 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:08,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:08,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:08,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:08,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:08,706 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:08,706 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:08,706 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:08,706 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:08,707 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,707 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,708 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:08,708 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:08,708 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,708 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:08,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:08,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:08,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:08,709 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:08,709 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:08,709 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:08,709 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:08,710 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,710 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:08,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1444156d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:08,712 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:08,712 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,712 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,712 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:08,712 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:08,712 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,712 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:08,712 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:08,713 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:08,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:08,713 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:08,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:08,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:08,715 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:08,715 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:08,715 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:08,715 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:08,715 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:08,716 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,716 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,716 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,716 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,718 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,728 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,728 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:08,728 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:08,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:08,728 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:08,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:08,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:08,728 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:08,728 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:08,728 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:08,728 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,729 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:08,729 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:08,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:08,729 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,729 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530408760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530408762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530408764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,829 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:08,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:08,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:08,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:08,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:08,831 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:08,831 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:08,831 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:08,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:08,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,832 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,832 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:08,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:08,833 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,833 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:08,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:08,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:08,834 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:08,834 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:08,834 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:08,834 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:08,834 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:08,835 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,835 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:08,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395bf53a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:08,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:08,836 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,837 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:08,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:08,838 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:08,838 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,838 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:08,839 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,839 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,839 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:08,839 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:08,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:08,840 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:08,840 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:08,840 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:08,840 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:08,841 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:08,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,841 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:08,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:08,842 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,843 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,844 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:08,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:08,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:08,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:08,844 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:08,844 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:08,844 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:08,944 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:08,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:08,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:08,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:08,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:08,947 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:08,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:08,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:08,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:08,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,947 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,948 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,948 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:08,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:08,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:08,948 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,948 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:08,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:08,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:08,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:08,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:08,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:08,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:08,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:08,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:08,949 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:08,949 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:08,949 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:08,949 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:08,949 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:08,950 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:08,950 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:08,950 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,950 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:08,950 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:08,950 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,950 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,951 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:08,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b971f5d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:08,952 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:08,952 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,952 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,952 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:08,952 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,953 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:08,952 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:08,953 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:08,953 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,953 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:08,953 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,953 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:08,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:08,955 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:08,955 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:08,955 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:08,955 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:08,955 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:08,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:08,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,956 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,956 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:08,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,957 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:08,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:08,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:08,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:08,958 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:08,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:08,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:08,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:08,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:08,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:08,958 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:08,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:08,959 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:08,959 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:08,959 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:08,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:08,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:08,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530408963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60746 deadline: 1680530408963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:08,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:08,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60822 deadline: 1680530408965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,059 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:09,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:09,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:09,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:09,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:09,062 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:09,062 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:09,062 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:09,062 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:09,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:09,063 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,064 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:09,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:09,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,064 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:09,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:09,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:09,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:09,065 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:09,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:09,065 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:09,065 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:09,065 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:09,065 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:09,065 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:09,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:09,066 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:09,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:09,066 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:09,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:09,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:09,067 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:09,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ce249c[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:09,067 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:09,068 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:09,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:09,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:09,068 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:09,068 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:09,068 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:09,068 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,068 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:09,068 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:09,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:09,069 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:09,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:09,070 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:09,070 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:09,070 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:09,070 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:09,070 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:09,071 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,071 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,071 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:09,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:09,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:09,071 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:09,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:09,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:09,072 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:09,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:09,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:09,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:09,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:09,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:09,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:09,073 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:09,073 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:09,074 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:09,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:09,073 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:09,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:09,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:09,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:09,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:09,074 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/eff8d7de2b42451ca5c78a81e675a399 2023-04-03 13:59:09,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=47.28 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/2054a29bc9be4096a05b520025788a97 2023-04-03 13:59:09,084 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/339ca567b9b44b9ca86516570f61874a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/339ca567b9b44b9ca86516570f61874a 2023-04-03 13:59:09,086 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/339ca567b9b44b9ca86516570f61874a, entries=150, sequenceid=268, filesize=25.4 K 2023-04-03 13:59:09,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/eff8d7de2b42451ca5c78a81e675a399 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/eff8d7de2b42451ca5c78a81e675a399 2023-04-03 13:59:09,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/eff8d7de2b42451ca5c78a81e675a399, entries=150, sequenceid=268, filesize=11.9 K 2023-04-03 13:59:09,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/2054a29bc9be4096a05b520025788a97 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2054a29bc9be4096a05b520025788a97 2023-04-03 13:59:09,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2054a29bc9be4096a05b520025788a97, entries=150, sequenceid=268, filesize=11.9 K 2023-04-03 13:59:09,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~145.68 KB/149172, heapSize ~416.43 KB/426424, currentSize=33.54 KB/34350 for af4327b273d03b2199ab4cc1b2fa1539 in 440ms, sequenceid=268, compaction requested=true 2023-04-03 13:59:09,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:09,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:09,094 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:59:09,094 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:59:09,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:09,095 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:09,096 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 155732 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:59:09,096 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/A is initiating minor compaction (all files) 2023-04-03 13:59:09,096 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/A in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:09,096 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/93b854ec6e674fdd8400f993de3318b1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d1299f452cbe4571852af221ddb5c689, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa23d20d85434d2eac5d028b314a64c0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/ca2a3a24f68a4116a7e1489be6c190cb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/dc9de894e9ea45f4a7c29c4ad08c7ab8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/339ca567b9b44b9ca86516570f61874a] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=152.1 K 2023-04-03 13:59:09,097 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72572 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:59:09,097 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/B is initiating minor compaction (all files) 2023-04-03 13:59:09,097 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/B in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:09,097 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d99472743cd04bd4aad954388ceb2640, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/293a2e65aafb4e768563c52a55a50a94, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11ab2c30f3a546c096344d239b900c2f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/003f26533cd74e699a91b07f2417e10c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/52827c2b03fb49b4bba3fc4c23ade9ad, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/eff8d7de2b42451ca5c78a81e675a399] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=70.9 K 2023-04-03 13:59:09,098 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 93b854ec6e674fdd8400f993de3318b1, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530341653 2023-04-03 13:59:09,098 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d99472743cd04bd4aad954388ceb2640, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530341653 2023-04-03 13:59:09,098 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting d1299f452cbe4571852af221ddb5c689, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680530342926 2023-04-03 13:59:09,099 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting aa23d20d85434d2eac5d028b314a64c0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680530343250 2023-04-03 13:59:09,099 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting ca2a3a24f68a4116a7e1489be6c190cb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680530344723 2023-04-03 13:59:09,099 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 293a2e65aafb4e768563c52a55a50a94, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680530342926 2023-04-03 13:59:09,100 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting dc9de894e9ea45f4a7c29c4ad08c7ab8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680530346857 2023-04-03 13:59:09,100 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 339ca567b9b44b9ca86516570f61874a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680530348037 2023-04-03 13:59:09,100 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 11ab2c30f3a546c096344d239b900c2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680530343250 2023-04-03 13:59:09,100 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 003f26533cd74e699a91b07f2417e10c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680530344723 2023-04-03 13:59:09,101 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 52827c2b03fb49b4bba3fc4c23ade9ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680530346857 2023-04-03 13:59:09,101 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting eff8d7de2b42451ca5c78a81e675a399, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680530348037 2023-04-03 13:59:09,111 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#B#compaction#478 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:09,114 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#A#compaction#477 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:09,174 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:09,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:09,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:09,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:09,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:09,177 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:09,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:09,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:09,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:09,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,177 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:09,178 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:09,178 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:09,178 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,178 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:09,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:09,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:09,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:09,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:09,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:09,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:09,179 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:09,179 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:09,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:09,180 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 13:59:09,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:09,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:09,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:09,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:09,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:09,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:09,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:09,278 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:09,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530349288, max timestamp=1680530349326Num uniques 859; 2023-04-03 13:59:09,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:09,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530349288, max timestamp=1680530349326Num uniques 859; 2023-04-03 13:59:09,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:09,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530349288, max timestamp=1680530349326Num uniques 859; 2023-04-03 13:59:09,328 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:09,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530349335, max timestamp=1680530349352Num uniques 1018; 2023-04-03 13:59:09,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530349335, max timestamp=1680530349352Num uniques 1018; 2023-04-03 13:59:09,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 13:59:09,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680530349335, max timestamp=1680530349352Num uniques 1018; 2023-04-03 13:59:09,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 13:59:09,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 13:59:09,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530409376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530349356, max timestamp=1680530349379Num uniques 1077; 2023-04-03 13:59:09,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-03 13:59:09,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530349356, max timestamp=1680530349379Num uniques 1077; 2023-04-03 13:59:09,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-03 13:59:09,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680530349356, max timestamp=1680530349379Num uniques 1077; 2023-04-03 13:59:09,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-04-03 13:59:09,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530409413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530409415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60806 deadline: 1680530409481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530409517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530409518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,526 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/2fb9248c8f054c6db7e3d39c52aa4f0a as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2fb9248c8f054c6db7e3d39c52aa4f0a 2023-04-03 13:59:09,530 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/B of af4327b273d03b2199ab4cc1b2fa1539 into 2fb9248c8f054c6db7e3d39c52aa4f0a(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:09,530 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:09,530 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/B, priority=10, startTime=1680530349094; duration=0sec 2023-04-03 13:59:09,530 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:09,531 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 13:59:09,532 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72572 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 13:59:09,532 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/C is initiating minor compaction (all files) 2023-04-03 13:59:09,532 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/C in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:09,532 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/86fe47bd118448c89e67bc6f29939eb1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7d45d23797b445ef9f2ffff5932d8006, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d37c77462b494696a8a97cf08d95073a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2770ca780f114f26837155a7b4099964, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d2d48e29fceb45908cb20bbb211d31f0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2054a29bc9be4096a05b520025788a97] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=70.9 K 2023-04-03 13:59:09,532 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 86fe47bd118448c89e67bc6f29939eb1, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1680530341653 2023-04-03 13:59:09,532 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 7d45d23797b445ef9f2ffff5932d8006, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680530342926 2023-04-03 13:59:09,533 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d37c77462b494696a8a97cf08d95073a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680530343250 2023-04-03 13:59:09,533 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/b3b506df1b154a4d93fd6a141442a3f5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/b3b506df1b154a4d93fd6a141442a3f5 2023-04-03 13:59:09,533 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2770ca780f114f26837155a7b4099964, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=221, earliestPutTs=1680530344723 2023-04-03 13:59:09,533 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d2d48e29fceb45908cb20bbb211d31f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1680530346857 2023-04-03 13:59:09,533 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2054a29bc9be4096a05b520025788a97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680530348037 2023-04-03 13:59:09,536 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/A of af4327b273d03b2199ab4cc1b2fa1539 into b3b506df1b154a4d93fd6a141442a3f5(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:09,536 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:09,536 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/A, priority=10, startTime=1680530349094; duration=0sec 2023-04-03 13:59:09,536 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:09,540 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#C#compaction#480 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:09,546 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/e07232af82e94c58950812efe5de7ed1 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e07232af82e94c58950812efe5de7ed1 2023-04-03 13:59:09,549 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/C of af4327b273d03b2199ab4cc1b2fa1539 into e07232af82e94c58950812efe5de7ed1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:09,549 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:09,549 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/C, priority=10, startTime=1680530349095; duration=0sec 2023-04-03 13:59:09,549 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:09,602 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403c709c0e0ce3b467daff60395065c0bb9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c709c0e0ce3b467daff60395065c0bb9 2023-04-03 13:59:09,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530409682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530409719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530409721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:09,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:09,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530409984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,006 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=276, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/31440f7c3c254b038c1907561d88ce7b 2023-04-03 13:59:10,013 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/fb90101d87384395909a254e27ccd9b4 2023-04-03 13:59:10,023 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=276 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/bc11fbef6db744409607cb0959819749 2023-04-03 13:59:10,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:10,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530410023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:10,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530410023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,026 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/31440f7c3c254b038c1907561d88ce7b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/31440f7c3c254b038c1907561d88ce7b 2023-04-03 13:59:10,028 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/31440f7c3c254b038c1907561d88ce7b, entries=150, sequenceid=276, filesize=25.4 K 2023-04-03 13:59:10,029 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/fb90101d87384395909a254e27ccd9b4 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/fb90101d87384395909a254e27ccd9b4 2023-04-03 13:59:10,032 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/fb90101d87384395909a254e27ccd9b4, entries=150, sequenceid=276, filesize=11.9 K 2023-04-03 13:59:10,032 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/bc11fbef6db744409607cb0959819749 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/bc11fbef6db744409607cb0959819749 2023-04-03 13:59:10,035 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/bc11fbef6db744409607cb0959819749, entries=150, sequenceid=276, filesize=11.9 K 2023-04-03 13:59:10,035 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for af4327b273d03b2199ab4cc1b2fa1539 in 855ms, sequenceid=276, compaction requested=false 2023-04-03 13:59:10,036 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:10,036 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:10,036 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:10,036 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:10,036 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,038 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:10,038 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:10,039 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,039 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:10,039 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:10,039 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:10,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:10,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:10,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:10,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3195b635[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:10,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:10,040 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:10,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:10,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:10,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:10,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:10,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:10,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:10,041 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,041 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:10,042 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:10,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:10,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:10,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:10,044 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:10,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:10,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:10,044 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:10,044 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:10,045 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,045 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:10,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,045 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:10,045 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:10,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:10,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:10,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:10,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:10,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:10,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:10,047 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:10,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:10,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:10,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:10,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:10,048 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:10,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:10,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:10,048 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:10,048 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:10,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:10,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:10,148 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:10,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:10,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:10,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:10,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:10,151 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:10,151 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:10,151 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:10,151 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:10,152 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,152 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:10,152 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:10,152 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:10,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:10,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:10,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:10,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:10,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:10,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:10,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:10,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:10,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:10,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:10,153 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:10,153 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:10,154 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:10,154 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=147.60 KB heapSize=413.77 KB 2023-04-03 13:59:10,154 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:10,154 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:10,154 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:10,154 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:10,154 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:10,154 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:10,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:10,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:10,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:10,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530410502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530410525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,527 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:10,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530410526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,571 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304030bff1940450944828fba1e4192c2666f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030bff1940450944828fba1e4192c2666f 2023-04-03 13:59:10,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:10,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530410605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:10,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530410807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:10,975 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=350, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/66aef0071d3f490093f967b1b1619f67 2023-04-03 13:59:10,982 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/bee0aee47e2d4b34aece936fa1d8e5de 2023-04-03 13:59:11,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530411109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,389 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=350 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/4d25f17f11e84fe39a0be1d3dcdc4269 2023-04-03 13:59:11,392 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/66aef0071d3f490093f967b1b1619f67 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/66aef0071d3f490093f967b1b1619f67 2023-04-03 13:59:11,395 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/66aef0071d3f490093f967b1b1619f67, entries=150, sequenceid=350, filesize=25.4 K 2023-04-03 13:59:11,396 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/bee0aee47e2d4b34aece936fa1d8e5de as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bee0aee47e2d4b34aece936fa1d8e5de 2023-04-03 13:59:11,398 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bee0aee47e2d4b34aece936fa1d8e5de, entries=150, sequenceid=350, filesize=11.9 K 2023-04-03 13:59:11,399 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/4d25f17f11e84fe39a0be1d3dcdc4269 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/4d25f17f11e84fe39a0be1d3dcdc4269 2023-04-03 13:59:11,401 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/4d25f17f11e84fe39a0be1d3dcdc4269, entries=150, sequenceid=350, filesize=11.9 K 2023-04-03 13:59:11,402 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~413.72 KB/423648, currentSize=33.54 KB/34350 for af4327b273d03b2199ab4cc1b2fa1539 in 1248ms, sequenceid=350, compaction requested=true 2023-04-03 13:59:11,402 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:11,402 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,402 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:11,402 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:11,402 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,402 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:11,402 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:11,403 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,403 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,403 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,403 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:11,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:11,404 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57817718[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:11,405 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:11,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:11,405 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:11,405 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,405 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:11,406 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:11,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,406 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:11,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:11,408 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:11,408 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:11,408 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:11,408 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:11,408 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:11,409 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,409 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,409 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,410 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,411 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:11,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,413 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:11,413 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:11,413 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,413 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,414 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,414 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,414 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:11,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:59:11,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:11,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:11,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:11,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:11,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:11,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:11,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530351459, max timestamp=1680530351474Num uniques 859; 2023-04-03 13:59:11,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:11,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530351459, max timestamp=1680530351474Num uniques 859; 2023-04-03 13:59:11,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530351459, max timestamp=1680530351474Num uniques 859; 2023-04-03 13:59:11,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:11,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:11,513 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:11,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:11,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:11,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:11,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:11,516 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:11,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:11,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:11,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,516 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,517 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,517 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:11,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:11,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:11,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:11,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,518 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:11,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:11,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:11,519 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,519 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:11,519 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:11,519 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:11,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:11,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:11,519 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:11,520 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,520 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,520 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,520 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:11,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43fe742[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:11,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:11,521 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,521 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:11,522 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,522 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:11,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:11,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:11,522 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,522 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:11,524 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:11,524 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:11,524 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:11,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:11,524 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:11,524 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:11,525 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,525 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,525 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,526 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,527 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:11,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:11,528 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,529 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,529 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,528 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60778 deadline: 1680530411531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530411532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530411536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530411613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,629 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:11,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:11,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:11,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:11,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:11,631 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:11,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:11,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:11,631 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,632 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,632 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:11,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:11,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:11,633 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:11,633 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,633 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:11,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:11,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:11,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:11,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:11,634 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:11,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:11,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:11,634 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:11,635 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,635 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60778 deadline: 1680530411634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,638 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:11,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530411636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:11,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70a1d99d[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:11,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:11,639 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,640 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:11,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,640 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:11,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:11,640 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:11,641 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,641 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,641 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,641 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:11,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:11,643 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:11,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:11,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:11,643 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:11,643 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:11,643 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,644 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,644 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,644 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,645 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:11,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:11,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:11,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,647 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,647 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:11,647 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,648 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,648 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:11,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,748 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:11,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:11,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:11,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:11,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:11,751 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:11,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:11,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:11,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,751 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,752 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,752 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:11,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:11,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:11,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:11,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:11,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:11,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:11,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:11,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:11,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:11,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:11,753 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:11,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:11,753 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,754 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:11,754 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:11,754 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,754 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,755 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:11,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:11,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c99fda2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:11,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,756 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,756 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:11,756 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:11,756 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:11,756 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,757 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:11,757 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:11,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,757 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:11,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:11,759 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:11,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:11,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:11,759 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:11,759 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:11,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,760 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,760 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,761 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,763 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,762 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:11,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:11,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,763 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,763 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:11,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:11,763 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:11,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60778 deadline: 1680530411837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,840 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304039e7042f365d14d3f9b63d2a069a08d2f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039e7042f365d14d3f9b63d2a069a08d2f 2023-04-03 13:59:11,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:11,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530411839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,863 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:11,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:11,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:11,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:11,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:11,866 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:11,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:11,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:11,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,866 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,867 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,867 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,867 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,867 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:11,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,868 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:11,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,868 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:11,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:11,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:11,869 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,869 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:11,869 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:11,869 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:11,869 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:11,870 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,870 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:11,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30fc6e9e[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:11,876 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:11,876 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,883 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:11,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,884 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:11,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:11,884 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:11,886 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,886 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,887 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:11,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:11,888 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:11,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:11,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:11,888 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:11,888 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:11,889 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,889 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,889 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:11,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:11,891 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:11,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,892 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,892 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:11,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:11,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:11,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:11,893 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:11,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,893 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:11,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:11,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:11,993 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:11,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:11,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:11,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:11,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:11,996 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:11,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:11,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:11,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,996 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,997 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,997 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:11,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:11,997 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,997 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:11,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:11,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:11,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:11,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:11,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:11,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:11,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:11,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:11,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:11,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:11,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:11,998 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,998 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:11,998 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:11,998 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:11,998 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:11,999 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:11,999 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,999 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:11,999 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:11,999 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,999 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:11,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,000 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,000 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,001 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b828a34[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,001 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,001 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,001 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,001 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,001 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,001 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,001 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,002 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,002 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,003 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,003 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,003 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,003 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,003 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,004 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,004 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,004 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,005 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,007 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,007 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,007 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,007 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,007 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,007 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:12,007 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,008 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,008 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,008 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,108 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:12,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:12,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:12,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:12,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:12,111 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:12,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:12,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:12,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,112 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,112 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:12,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:12,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:12,113 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,113 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:12,113 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:12,113 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:12,113 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:12,114 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,114 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,114 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,114 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3511cfe8[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,116 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,116 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,116 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,117 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,119 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,119 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,119 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,120 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,120 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,120 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,120 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,123 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,123 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,123 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,123 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,123 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,124 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,124 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,123 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:12,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60778 deadline: 1680530412140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:12,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530412144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,224 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:12,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:12,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:12,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:12,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:12,227 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:12,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:12,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:12,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,227 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,228 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,228 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,228 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,228 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:12,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:12,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:12,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:12,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:12,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:12,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:12,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:12,229 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:12,229 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,229 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:12,230 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:12,230 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:12,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:12,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:12,230 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:12,231 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,231 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,231 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16465005[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,232 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,233 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,233 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,233 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,233 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,233 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,233 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,233 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,233 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,233 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,236 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,236 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,236 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,236 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,236 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,237 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,237 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,237 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,237 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,239 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:12,241 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,241 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,241 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,241 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,245 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=360, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/d857bec3b3b64e95a7e7577a7cbf53f5 2023-04-03 13:59:12,341 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:12,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:12,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:12,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:12,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:12,344 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:12,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:12,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:12,344 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,345 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,345 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,345 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,345 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,345 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:12,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:12,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:12,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:12,346 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:12,346 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,346 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:12,346 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:12,346 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:12,346 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,346 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:12,346 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:12,347 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:12,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:12,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:12,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:12,347 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,347 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,348 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@459d44cd[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,348 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,348 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,349 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,349 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,349 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,349 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,350 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,350 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,352 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,352 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,352 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,352 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,353 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,354 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,356 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,356 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,356 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,356 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,456 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:12,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:12,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:12,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:12,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:12,459 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:12,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:12,459 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:12,460 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,460 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,461 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,461 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,461 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,461 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:12,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:12,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:12,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:12,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:12,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:12,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:12,462 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,462 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:12,462 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:12,462 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:12,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:12,462 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:12,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:12,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,463 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,463 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,464 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,465 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a91b560[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,465 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,465 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,465 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,465 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,465 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,466 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,466 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,467 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,467 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,469 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,469 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,469 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,469 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,470 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,471 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,472 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,472 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,472 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:12,472 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,473 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,473 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,473 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,473 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,573 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:12,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:12,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:12,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:12,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:12,575 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:12,575 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,575 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:12,575 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:12,575 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,575 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,576 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,576 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,577 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:12,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:12,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:12,577 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,577 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:12,577 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:12,577 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:12,577 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:12,578 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,578 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,578 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,578 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21444755[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,579 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,580 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,580 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,580 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,580 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,580 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,580 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,581 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,581 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,581 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,581 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,582 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,582 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,583 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,583 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,583 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,584 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,584 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:12,586 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,586 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,586 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,586 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:12,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530412618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:12,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60778 deadline: 1680530412645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,649 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:12,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530412648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/c09720af9cc549c0a55a8b06d16d8b73 2023-04-03 13:59:12,686 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:12,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:12,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:12,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:12,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:12,689 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:12,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:12,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:12,689 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,690 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,690 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,690 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,690 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,691 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:12,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:12,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:12,693 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,693 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:12,693 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:12,693 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:12,693 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:12,694 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,694 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,695 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a6fd00[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,695 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,696 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,696 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,696 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,696 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,696 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,696 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,697 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,697 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,699 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,699 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,699 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,699 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,699 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,700 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,700 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,700 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,700 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,701 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,702 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,702 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,703 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,703 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:12,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,703 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,803 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:12,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:12,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:12,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:12,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:12,806 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:12,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:12,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:12,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,806 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,807 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,807 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:12,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:12,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:12,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:12,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:12,808 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:12,808 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,808 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:12,809 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:12,809 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:12,809 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:12,810 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,810 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,810 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,811 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ab26a55[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,811 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,812 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,812 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,812 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,812 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,812 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,812 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,812 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,813 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,813 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,813 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,813 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,815 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,815 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,815 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,815 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,815 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,816 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,817 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,817 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,817 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,818 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:12,820 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,820 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,820 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,820 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,920 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:12,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:12,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:12,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:12,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:12,923 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:12,923 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,923 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:12,923 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:12,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,924 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,924 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,925 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,925 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:12,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:12,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:12,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:12,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:12,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:12,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:12,925 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:12,926 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,926 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:12,926 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:12,926 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:12,926 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:12,926 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:12,926 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:12,926 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,926 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:12,926 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:12,927 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,927 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:12,928 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:12,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d9ff0d6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:12,928 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,928 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,929 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:12,929 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:12,929 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:12,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,929 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:12,929 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:12,929 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,929 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,929 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:12,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:12,931 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:12,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:12,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:12,931 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:12,931 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:12,932 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,932 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:12,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,932 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:12,932 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,933 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:12,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:12,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:12,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:12,935 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:12,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:12,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:12,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:12,935 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:12,934 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:12,935 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,935 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:12,935 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:12,935 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:12,935 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:12,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:12,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:13,035 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:13,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:13,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:13,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:13,038 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:13,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:13,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:13,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:13,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,038 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:13,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,039 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:13,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:13,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,039 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:13,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:13,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:13,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:13,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:13,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:13,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:13,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:13,040 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:13,040 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:13,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:13,041 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:13,041 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:13,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:13,042 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:13,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:13,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ee276df[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:13,043 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:13,043 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:13,043 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:13,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:13,043 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:13,043 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,043 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:13,044 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:13,044 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:13,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:13,046 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:13,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:13,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:13,046 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:13,047 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:13,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,047 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:13,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,048 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:13,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:13,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:13,048 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,050 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:13,050 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:13,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:13,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:13,050 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:13,050 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:13,050 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:13,051 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:13,051 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:13,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:13,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:13,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:13,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:13,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:13,051 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/ec28eb8a88124ea292989d80bb594508 2023-04-03 13:59:13,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/d857bec3b3b64e95a7e7577a7cbf53f5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d857bec3b3b64e95a7e7577a7cbf53f5 2023-04-03 13:59:13,069 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d857bec3b3b64e95a7e7577a7cbf53f5, entries=200, sequenceid=360, filesize=32.3 K 2023-04-03 13:59:13,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/c09720af9cc549c0a55a8b06d16d8b73 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/c09720af9cc549c0a55a8b06d16d8b73 2023-04-03 13:59:13,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/c09720af9cc549c0a55a8b06d16d8b73, entries=150, sequenceid=360, filesize=11.9 K 2023-04-03 13:59:13,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/ec28eb8a88124ea292989d80bb594508 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ec28eb8a88124ea292989d80bb594508 2023-04-03 13:59:13,075 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ec28eb8a88124ea292989d80bb594508, entries=150, sequenceid=360, filesize=11.9 K 2023-04-03 13:59:13,076 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for af4327b273d03b2199ab4cc1b2fa1539 in 1645ms, sequenceid=360, compaction requested=true 2023-04-03 13:59:13,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:13,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:13,076 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:13,076 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:13,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:13,076 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:13,077 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111634 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:13,077 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:13,077 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/B is initiating minor compaction (all files) 2023-04-03 13:59:13,077 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/B in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:13,077 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2fb9248c8f054c6db7e3d39c52aa4f0a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/fb90101d87384395909a254e27ccd9b4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bee0aee47e2d4b34aece936fa1d8e5de, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/c09720af9cc549c0a55a8b06d16d8b73] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=48.0 K 2023-04-03 13:59:13,077 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/A is initiating minor compaction (all files) 2023-04-03 13:59:13,077 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/A in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:13,077 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/b3b506df1b154a4d93fd6a141442a3f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/31440f7c3c254b038c1907561d88ce7b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/66aef0071d3f490093f967b1b1619f67, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d857bec3b3b64e95a7e7577a7cbf53f5] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=109.0 K 2023-04-03 13:59:13,078 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 2fb9248c8f054c6db7e3d39c52aa4f0a, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680530348037 2023-04-03 13:59:13,078 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting b3b506df1b154a4d93fd6a141442a3f5, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680530348037 2023-04-03 13:59:13,078 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting fb90101d87384395909a254e27ccd9b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680530348656 2023-04-03 13:59:13,078 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 31440f7c3c254b038c1907561d88ce7b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680530348656 2023-04-03 13:59:13,078 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bee0aee47e2d4b34aece936fa1d8e5de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680530349394 2023-04-03 13:59:13,078 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 66aef0071d3f490093f967b1b1619f67, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680530349394 2023-04-03 13:59:13,079 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c09720af9cc549c0a55a8b06d16d8b73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680530350492 2023-04-03 13:59:13,079 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting d857bec3b3b64e95a7e7577a7cbf53f5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680530350492 2023-04-03 13:59:13,086 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#B#compaction#489 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:13,088 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#A#compaction#490 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:13,103 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/1305b9eea8c24754ae49fef2971aaf34 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1305b9eea8c24754ae49fef2971aaf34 2023-04-03 13:59:13,107 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/B of af4327b273d03b2199ab4cc1b2fa1539 into 1305b9eea8c24754ae49fef2971aaf34(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:13,107 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:13,107 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/B, priority=12, startTime=1680530353076; duration=0sec 2023-04-03 13:59:13,107 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:13,107 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:13,108 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49154 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:13,108 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/C is initiating minor compaction (all files) 2023-04-03 13:59:13,108 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/C in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:13,108 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e07232af82e94c58950812efe5de7ed1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/bc11fbef6db744409607cb0959819749, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/4d25f17f11e84fe39a0be1d3dcdc4269, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ec28eb8a88124ea292989d80bb594508] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=48.0 K 2023-04-03 13:59:13,109 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e07232af82e94c58950812efe5de7ed1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1680530348037 2023-04-03 13:59:13,109 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting bc11fbef6db744409607cb0959819749, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=276, earliestPutTs=1680530348656 2023-04-03 13:59:13,109 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 4d25f17f11e84fe39a0be1d3dcdc4269, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=350, earliestPutTs=1680530349394 2023-04-03 13:59:13,109 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ec28eb8a88124ea292989d80bb594508, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680530350492 2023-04-03 13:59:13,114 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/c6534b50d6bf4c668911ee36dcee31ef as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c6534b50d6bf4c668911ee36dcee31ef 2023-04-03 13:59:13,115 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#C#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:13,119 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/A of af4327b273d03b2199ab4cc1b2fa1539 into c6534b50d6bf4c668911ee36dcee31ef(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:13,119 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:13,119 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/A, priority=12, startTime=1680530353076; duration=0sec 2023-04-03 13:59:13,119 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:13,122 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/ed50dcb195164c6da8e6b83b22b53941 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ed50dcb195164c6da8e6b83b22b53941 2023-04-03 13:59:13,126 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/C of af4327b273d03b2199ab4cc1b2fa1539 into ed50dcb195164c6da8e6b83b22b53941(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:13,126 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:13,126 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/C, priority=12, startTime=1680530353076; duration=0sec 2023-04-03 13:59:13,126 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:13,151 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:13,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:13,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:13,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:13,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:13,153 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:13,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:13,153 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:13,154 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:13,154 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,154 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:13,154 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:13,154 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:13,155 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,155 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:13,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:13,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:13,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:13,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:13,156 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:13,156 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:13,156 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:13,156 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:13,156 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:13,156 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:13,156 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:13,156 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 13:59:13,156 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:13,156 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:13,157 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:13,157 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:13,157 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:13,157 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:13,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:13,548 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:13,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:13,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530413565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,570 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304035feb17a252474ff59c4a74bf6e6af58c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035feb17a252474ff59c4a74bf6e6af58c 2023-04-03 13:59:13,574 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=400, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/0982582878724f23950726eafd2905c5 2023-04-03 13:59:13,583 WARN [DataStreamer for file /user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/e3e2b94ce885417fb24caa5b50e9757d] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-03 13:59:13,586 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/e3e2b94ce885417fb24caa5b50e9757d 2023-04-03 13:59:13,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:13,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60778 deadline: 1680530413647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:13,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530413657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:13,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530413667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:13,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530413871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:13,994 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/f409162096254a9cbf1177f924fa2a17 2023-04-03 13:59:13,997 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/0982582878724f23950726eafd2905c5 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/0982582878724f23950726eafd2905c5 2023-04-03 13:59:14,002 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/0982582878724f23950726eafd2905c5, entries=150, sequenceid=400, filesize=25.4 K 2023-04-03 13:59:14,003 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/e3e2b94ce885417fb24caa5b50e9757d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e3e2b94ce885417fb24caa5b50e9757d 2023-04-03 13:59:14,006 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e3e2b94ce885417fb24caa5b50e9757d, entries=150, sequenceid=400, filesize=11.9 K 2023-04-03 13:59:14,006 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/f409162096254a9cbf1177f924fa2a17 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f409162096254a9cbf1177f924fa2a17 2023-04-03 13:59:14,009 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f409162096254a9cbf1177f924fa2a17, entries=150, sequenceid=400, filesize=11.9 K 2023-04-03 13:59:14,009 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for af4327b273d03b2199ab4cc1b2fa1539 in 853ms, sequenceid=400, compaction requested=false 2023-04-03 13:59:14,010 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:14,010 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:14,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:14,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:14,010 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:14,011 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,011 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,011 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,011 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:14,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9ed2fe[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:14,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:14,012 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,013 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,013 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,013 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:14,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,013 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:14,013 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:14,013 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:14,014 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,014 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,014 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:14,015 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:14,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:14,016 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:14,017 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:14,017 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:14,017 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:14,017 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:14,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,017 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,018 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:14,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,018 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,019 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:14,021 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:14,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:14,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:14,021 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,021 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:14,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:14,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,021 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,121 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:14,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:14,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:14,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:14,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:14,124 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:14,124 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:14,124 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:14,124 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:14,124 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,124 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,125 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:14,125 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,125 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,125 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:14,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:14,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:14,126 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:14,126 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,126 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:14,126 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:59:14,126 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:14,126 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:14,127 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:14,127 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:14,127 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:14,127 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:14,142 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a87f43214a7843129e8caf8a58e29936 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a87f43214a7843129e8caf8a58e29936 2023-04-03 13:59:14,147 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=410, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/eb4e9125f4bc481ea8349bc17a30ae9d 2023-04-03 13:59:14,157 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/075f034f93ac47d6b595e1c0d18abd72 2023-04-03 13:59:14,166 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/c73e7cd620f34433b1ae4c6fbef896d0 2023-04-03 13:59:14,170 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/eb4e9125f4bc481ea8349bc17a30ae9d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/eb4e9125f4bc481ea8349bc17a30ae9d 2023-04-03 13:59:14,172 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/eb4e9125f4bc481ea8349bc17a30ae9d, entries=100, sequenceid=410, filesize=18.6 K 2023-04-03 13:59:14,173 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/075f034f93ac47d6b595e1c0d18abd72 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/075f034f93ac47d6b595e1c0d18abd72 2023-04-03 13:59:14,176 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/075f034f93ac47d6b595e1c0d18abd72, entries=100, sequenceid=410, filesize=9.5 K 2023-04-03 13:59:14,176 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/c73e7cd620f34433b1ae4c6fbef896d0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/c73e7cd620f34433b1ae4c6fbef896d0 2023-04-03 13:59:14,179 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/c73e7cd620f34433b1ae4c6fbef896d0, entries=100, sequenceid=410, filesize=9.5 K 2023-04-03 13:59:14,180 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for af4327b273d03b2199ab4cc1b2fa1539 in 53ms, sequenceid=410, compaction requested=true 2023-04-03 13:59:14,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:14,180 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,180 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:14,180 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:14,180 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,180 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:14,180 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:14,181 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,181 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,181 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,181 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:14,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3547e1fa[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:14,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:14,183 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,183 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,183 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:14,183 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:14,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:14,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,184 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:14,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:14,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:14,184 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,184 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:14,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:14,186 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:14,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:14,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:14,187 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:14,187 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:14,187 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,188 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,188 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:14,188 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,189 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:14,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:14,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:14,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:14,191 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:14,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:14,191 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,191 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:14,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,191 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:14,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:59:14,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:14,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:14,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:14,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:14,250 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:14,250 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:14,291 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:14,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:14,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:14,296 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403e49475e982ee4c7e83fe78bdc87d8816 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e49475e982ee4c7e83fe78bdc87d8816 2023-04-03 13:59:14,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:14,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:14,302 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:14,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:14,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:14,302 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:14,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,303 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:14,303 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:14,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:14,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,304 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:14,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:14,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530354284, max timestamp=1680530354312Num uniques 859; 2023-04-03 13:59:14,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530354284, max timestamp=1680530354312Num uniques 859; 2023-04-03 13:59:14,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:14,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:14,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530354284, max timestamp=1680530354312Num uniques 859; 2023-04-03 13:59:14,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:14,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:14,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:14,315 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:14,316 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,316 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:14,316 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:14,316 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:14,316 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:14,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:14,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@771caec4[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:14,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:14,319 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,320 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:14,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:14,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:14,320 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:14,322 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:14,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:14,322 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:14,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:14,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:14,325 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:14,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:14,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:14,325 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:14,325 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:14,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,326 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:14,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:14,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:14,327 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:14,328 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:14,330 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:14,337 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:14,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:14,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:14,337 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:14,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:14,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:14,337 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:14,337 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:14,340 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=420, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/aa8d5fed29c645c3b05e0cf853de6a9c 2023-04-03 13:59:14,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/d7ae33a847df4c5aae38ad90dd58ee30 2023-04-03 13:59:14,367 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/89c608c193c14451a9666be72ef3c2a0 2023-04-03 13:59:14,370 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/aa8d5fed29c645c3b05e0cf853de6a9c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa8d5fed29c645c3b05e0cf853de6a9c 2023-04-03 13:59:14,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa8d5fed29c645c3b05e0cf853de6a9c, entries=200, sequenceid=420, filesize=32.3 K 2023-04-03 13:59:14,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/d7ae33a847df4c5aae38ad90dd58ee30 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d7ae33a847df4c5aae38ad90dd58ee30 2023-04-03 13:59:14,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d7ae33a847df4c5aae38ad90dd58ee30, entries=150, sequenceid=420, filesize=11.9 K 2023-04-03 13:59:14,377 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/89c608c193c14451a9666be72ef3c2a0 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/89c608c193c14451a9666be72ef3c2a0 2023-04-03 13:59:14,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:14,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530414375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/89c608c193c14451a9666be72ef3c2a0, entries=150, sequenceid=420, filesize=11.9 K 2023-04-03 13:59:14,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for af4327b273d03b2199ab4cc1b2fa1539 in 132ms, sequenceid=420, compaction requested=true 2023-04-03 13:59:14,381 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:14,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:14,381 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:14,381 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:14,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:14,381 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:14,382 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104722 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:14,382 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/A is initiating minor compaction (all files) 2023-04-03 13:59:14,382 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/A in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,382 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c6534b50d6bf4c668911ee36dcee31ef, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/0982582878724f23950726eafd2905c5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/eb4e9125f4bc481ea8349bc17a30ae9d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa8d5fed29c645c3b05e0cf853de6a9c] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=102.3 K 2023-04-03 13:59:14,383 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:14,383 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/B is initiating minor compaction (all files) 2023-04-03 13:59:14,383 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/B in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,383 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1305b9eea8c24754ae49fef2971aaf34, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e3e2b94ce885417fb24caa5b50e9757d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/075f034f93ac47d6b595e1c0d18abd72, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d7ae33a847df4c5aae38ad90dd58ee30] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=45.7 K 2023-04-03 13:59:14,383 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting c6534b50d6bf4c668911ee36dcee31ef, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680530350492 2023-04-03 13:59:14,383 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1305b9eea8c24754ae49fef2971aaf34, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680530350492 2023-04-03 13:59:14,383 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 0982582878724f23950726eafd2905c5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680530351517 2023-04-03 13:59:14,383 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting e3e2b94ce885417fb24caa5b50e9757d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680530351517 2023-04-03 13:59:14,384 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting eb4e9125f4bc481ea8349bc17a30ae9d, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680530353560 2023-04-03 13:59:14,384 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 075f034f93ac47d6b595e1c0d18abd72, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680530353560 2023-04-03 13:59:14,384 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting aa8d5fed29c645c3b05e0cf853de6a9c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680530354219 2023-04-03 13:59:14,384 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting d7ae33a847df4c5aae38ad90dd58ee30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680530354219 2023-04-03 13:59:14,389 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#B#compaction#502 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:14,408 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#A#compaction#501 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:14,422 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/afc57012d6a646b785789817e0f79616 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/afc57012d6a646b785789817e0f79616 2023-04-03 13:59:14,426 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/B of af4327b273d03b2199ab4cc1b2fa1539 into afc57012d6a646b785789817e0f79616(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:14,426 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:14,426 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/B, priority=12, startTime=1680530354381; duration=0sec 2023-04-03 13:59:14,426 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:14,427 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:14,428 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:14,428 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/C is initiating minor compaction (all files) 2023-04-03 13:59:14,428 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/C in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,428 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ed50dcb195164c6da8e6b83b22b53941, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f409162096254a9cbf1177f924fa2a17, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/c73e7cd620f34433b1ae4c6fbef896d0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/89c608c193c14451a9666be72ef3c2a0] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=45.7 K 2023-04-03 13:59:14,429 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting ed50dcb195164c6da8e6b83b22b53941, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680530350492 2023-04-03 13:59:14,429 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f409162096254a9cbf1177f924fa2a17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680530351517 2023-04-03 13:59:14,429 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting c73e7cd620f34433b1ae4c6fbef896d0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1680530353560 2023-04-03 13:59:14,429 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 89c608c193c14451a9666be72ef3c2a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680530354219 2023-04-03 13:59:14,434 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#C#compaction#503 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:14,438 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:14,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:14,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:14,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:14,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:14,440 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:14,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:14,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:14,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:14,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:14,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:14,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,441 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:14,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:14,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:14,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:14,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:14,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:14,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:14,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:14,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:14,442 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:14,442 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:14,442 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:14,442 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 13:59:14,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:14,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:14,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:14,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:14,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:14,443 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 13:59:14,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:14,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:14,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:14,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530414500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:14,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530414603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:14,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60806 deadline: 1680530414622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,625 DEBUG [Thread-3433] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4122 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:59:14,807 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:14,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530414806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:14,836 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/4417ee30e156410d96ad469e514fe860 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/4417ee30e156410d96ad469e514fe860 2023-04-03 13:59:14,839 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/A of af4327b273d03b2199ab4cc1b2fa1539 into 4417ee30e156410d96ad469e514fe860(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:14,839 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:14,839 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/A, priority=12, startTime=1680530354381; duration=0sec 2023-04-03 13:59:14,839 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:14,841 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/90b0c0072a5a4e18b1e7caa5846e4ccc as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/90b0c0072a5a4e18b1e7caa5846e4ccc 2023-04-03 13:59:14,844 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/C of af4327b273d03b2199ab4cc1b2fa1539 into 90b0c0072a5a4e18b1e7caa5846e4ccc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:14,844 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:14,844 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/C, priority=12, startTime=1680530354381; duration=0sec 2023-04-03 13:59:14,845 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:14,864 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403733a6d7bec6246548ebb9fb66d7003c9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403733a6d7bec6246548ebb9fb66d7003c9 2023-04-03 13:59:15,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:15,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530415110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:15,269 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=458, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/a9e5eecba9b64f52802846bf1e931d88 2023-04-03 13:59:15,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:15,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530415615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:15,661 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:15,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60778 deadline: 1680530415659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:15,662 DEBUG [Thread-3435] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:59:15,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:15,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60822 deadline: 1680530415667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:15,669 DEBUG [Thread-3431] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., hostname=jenkins-hbase20.apache.org,39695,1680530174856, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 13:59:15,676 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/9d455ddbc897438294e51ee71f6c4fa2 2023-04-03 13:59:16,084 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/1ba5bb55e4e343588bbfd04948b37256 2023-04-03 13:59:16,087 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/a9e5eecba9b64f52802846bf1e931d88 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/a9e5eecba9b64f52802846bf1e931d88 2023-04-03 13:59:16,090 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/a9e5eecba9b64f52802846bf1e931d88, entries=150, sequenceid=458, filesize=25.4 K 2023-04-03 13:59:16,090 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/9d455ddbc897438294e51ee71f6c4fa2 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/9d455ddbc897438294e51ee71f6c4fa2 2023-04-03 13:59:16,093 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/9d455ddbc897438294e51ee71f6c4fa2, entries=150, sequenceid=458, filesize=11.9 K 2023-04-03 13:59:16,093 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/1ba5bb55e4e343588bbfd04948b37256 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1ba5bb55e4e343588bbfd04948b37256 2023-04-03 13:59:16,097 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1ba5bb55e4e343588bbfd04948b37256, entries=150, sequenceid=458, filesize=11.9 K 2023-04-03 13:59:16,097 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for af4327b273d03b2199ab4cc1b2fa1539 in 1655ms, sequenceid=458, compaction requested=false 2023-04-03 13:59:16,097 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:16,097 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:16,097 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:16,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:16,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:16,098 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:16,098 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:16,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:16,099 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:16,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:16,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:16,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:16,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae56a76[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:16,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:16,111 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:16,117 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:16,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:16,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:16,117 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,117 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:16,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:16,117 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:16,118 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:16,118 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,118 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:16,118 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:16,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:16,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:16,122 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:16,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:16,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:16,122 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:16,122 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:16,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:16,124 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:16,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:16,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:16,125 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:16,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:16,126 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:16,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:16,127 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:16,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:16,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:16,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:16,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:16,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:16,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:16,127 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:16,128 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:16,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:16,128 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:16,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:16,128 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:16,228 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:16,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:16,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:16,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:16,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:16,230 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:16,230 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:16,230 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:16,230 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:16,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:16,231 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:16,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:16,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:16,231 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,232 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:16,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:16,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:16,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:16,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:16,232 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:16,232 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:16,232 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:16,233 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:16,233 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:16,233 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:16,233 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:16,233 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:59:16,233 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:16,233 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:16,233 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:16,233 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:16,233 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:16,233 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:16,298 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-03 13:59:16,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:16,621 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:16,650 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036959ba73039441878d62de8a805f8691 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036959ba73039441878d62de8a805f8691 2023-04-03 13:59:16,654 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=470, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/354067dcccfc4d7b8241e1559a09eb91 2023-04-03 13:59:16,661 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/1a42f595f038415ba074d681a761605c 2023-04-03 13:59:16,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530356641, max timestamp=1680530356666Num uniques 859; 2023-04-03 13:59:16,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:16,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530356641, max timestamp=1680530356666Num uniques 859; 2023-04-03 13:59:16,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680530356641, max timestamp=1680530356666Num uniques 859; 2023-04-03 13:59:16,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:16,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 13:59:16,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:16,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530416703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:16,807 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530416806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:17,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39695] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530417010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,071 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/f5b095c8840142248b5318d855a63e64 2023-04-03 13:59:17,074 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/354067dcccfc4d7b8241e1559a09eb91 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/354067dcccfc4d7b8241e1559a09eb91 2023-04-03 13:59:17,077 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/354067dcccfc4d7b8241e1559a09eb91, entries=150, sequenceid=470, filesize=25.4 K 2023-04-03 13:59:17,077 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/1a42f595f038415ba074d681a761605c as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1a42f595f038415ba074d681a761605c 2023-04-03 13:59:17,080 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1a42f595f038415ba074d681a761605c, entries=150, sequenceid=470, filesize=11.9 K 2023-04-03 13:59:17,080 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/f5b095c8840142248b5318d855a63e64 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f5b095c8840142248b5318d855a63e64 2023-04-03 13:59:17,083 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f5b095c8840142248b5318d855a63e64, entries=150, sequenceid=470, filesize=11.9 K 2023-04-03 13:59:17,083 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for af4327b273d03b2199ab4cc1b2fa1539 in 850ms, sequenceid=470, compaction requested=true 2023-04-03 13:59:17,083 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:17,084 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,084 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:17,084 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:17,084 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,084 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:17,084 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:17,085 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,085 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:17,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20964af6[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:17,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:17,086 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,086 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:17,086 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:17,087 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,087 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:17,087 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,087 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:17,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:17,089 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:17,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:17,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:17,089 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:17,089 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:17,090 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,090 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,090 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,090 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,091 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,093 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:17,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:17,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:17,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,093 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,093 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,093 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:17,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-03 13:59:17,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:17,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:17,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:17,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:17,116 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:17,116 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 13:59:17,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:17,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530417146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,194 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:17,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:17,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:17,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:17,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:17,196 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:17,196 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,196 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:17,196 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:17,196 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,197 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,197 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,197 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,197 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,197 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:17,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:17,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:17,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:17,198 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:17,198 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,198 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:17,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:17,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:17,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:17,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:17,199 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:17,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:17,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:17,199 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:17,200 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,200 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,200 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,200 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:17,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a54372f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:17,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:17,201 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,201 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:17,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:17,201 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:17,202 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,202 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:17,202 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,202 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:17,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,203 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:17,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:17,204 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:17,204 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:17,204 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:17,204 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:17,204 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:17,205 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,205 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,205 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,206 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,207 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,207 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,207 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,208 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,208 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:17,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:17,208 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:17,208 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:17,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:17,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530417249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,308 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:17,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:17,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:17,311 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:17,311 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,311 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:17,311 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:17,311 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,311 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,312 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,312 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,312 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,312 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:17,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:17,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:17,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:17,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:17,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:17,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:17,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:17,313 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:17,313 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,314 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:17,314 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:17,314 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:17,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:17,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:17,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530417312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:17,314 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:17,315 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,315 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,315 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,315 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:17,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@382cc62f[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:17,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:17,316 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,317 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:17,317 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:17,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:17,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:17,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,318 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:17,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:17,320 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:17,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:17,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:17,320 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:17,320 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:17,321 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,321 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,321 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,321 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,322 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,324 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:17,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,324 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,325 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:17,325 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:17,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:17,325 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,325 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,325 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,325 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,325 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,425 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:17,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:17,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:17,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:17,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:17,428 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:17,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:17,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:17,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,428 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,429 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,429 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,429 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,429 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:17,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:17,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:17,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:17,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:17,430 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,430 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:17,430 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:17,430 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:17,430 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,431 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:17,431 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:17,431 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,431 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,431 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,431 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:17,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:17,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49ebbd86[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:17,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:17,433 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,434 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,434 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:17,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:17,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:17,434 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:17,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:17,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,435 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,435 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:17,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:17,437 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:17,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:17,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:17,437 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:17,437 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:17,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,438 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,438 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,439 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,440 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,440 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,440 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,440 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:17,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:17,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,441 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:17,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,441 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:17,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,441 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:17,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530417453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,526 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403510034bdd30e46fca69987c400fcda59 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403510034bdd30e46fca69987c400fcda59 2023-04-03 13:59:17,530 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=508, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/df660e18e79d46deac5f502a1bfcab5d 2023-04-03 13:59:17,541 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:17,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:17,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:17,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:17,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:17,543 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:17,543 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,543 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:17,543 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:17,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,544 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,544 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,545 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,545 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:17,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:17,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:17,546 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,546 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:17,546 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:17,546 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:17,546 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:17,547 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,548 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,548 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,548 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:17,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bb89fb7[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:17,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:17,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,549 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,549 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,550 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:17,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:17,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:17,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:17,550 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,550 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:17,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:17,552 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:17,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:17,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:17,552 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:17,552 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:17,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,553 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,553 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,554 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,556 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:17,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,556 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,556 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,556 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,656 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:17,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:17,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:17,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:17,659 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:17,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:17,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:17,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,659 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,660 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,660 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,660 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,660 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:17,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:17,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:17,661 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:17,661 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,662 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:17,662 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:17,662 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:17,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:17,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:17,662 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:17,663 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,663 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:17,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@505e36a0[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:17,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:17,664 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,665 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:17,665 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,665 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:17,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:17,665 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:17,665 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,666 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:17,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:17,667 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:17,667 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:17,667 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:17,667 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:17,667 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:17,668 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,668 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,668 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,669 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,671 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,672 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:17,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:17,672 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:17,672 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:17,674 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,674 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,674 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,674 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,674 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,674 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,674 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,674 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:17,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:17,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39695] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:60750 deadline: 1680530417756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,772 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:17,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:17,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:17,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:17,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:17,775 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:17,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:17,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:17,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,775 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,776 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,776 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:17,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:17,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:17,777 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,777 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:17,777 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:17,777 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:17,777 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:17,778 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,778 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:17,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e9d9f3a[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:17,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:17,779 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:17,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:17,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:17,780 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,780 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:17,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:17,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,780 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,780 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:17,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:17,782 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:17,782 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:17,782 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:17,782 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:17,782 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:17,783 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,783 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,783 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,783 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,784 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,786 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:17,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,786 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,786 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:17,786 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:17,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 13:59:17,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:60746 deadline: 1680530417819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,886 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:17,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:17,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:17,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:17,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:17,889 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:17,889 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,889 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:17,889 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:17,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,890 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,890 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,891 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,891 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:17,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,891 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:17,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:17,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:17,892 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,892 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:17,892 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:17,892 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:17,892 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:17,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,893 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,893 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:17,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a0e04e2[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:17,894 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:17,894 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,895 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,895 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:17,895 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:17,895 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:17,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,895 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:17,896 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,896 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:17,896 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,896 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:17,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:17,898 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:17,898 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:17,898 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:17,898 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:17,898 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:17,899 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,899 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:17,899 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:17,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,899 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,900 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,902 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:17,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:17,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:17,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:17,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:17,902 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:17,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:17,902 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:17,903 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:17,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/64e3ed048b07427fbcb3cd214c2cca98 2023-04-03 13:59:17,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=508 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/1e493784e4694f32920153aea84c39e9 2023-04-03 13:59:17,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/df660e18e79d46deac5f502a1bfcab5d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/df660e18e79d46deac5f502a1bfcab5d 2023-04-03 13:59:17,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/df660e18e79d46deac5f502a1bfcab5d, entries=150, sequenceid=508, filesize=25.4 K 2023-04-03 13:59:17,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/64e3ed048b07427fbcb3cd214c2cca98 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/64e3ed048b07427fbcb3cd214c2cca98 2023-04-03 13:59:17,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/64e3ed048b07427fbcb3cd214c2cca98, entries=150, sequenceid=508, filesize=11.9 K 2023-04-03 13:59:17,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/1e493784e4694f32920153aea84c39e9 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1e493784e4694f32920153aea84c39e9 2023-04-03 13:59:17,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1e493784e4694f32920153aea84c39e9, entries=150, sequenceid=508, filesize=11.9 K 2023-04-03 13:59:17,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for af4327b273d03b2199ab4cc1b2fa1539 in 844ms, sequenceid=508, compaction requested=true 2023-04-03 13:59:17,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:17,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:17,959 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:17,960 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:17,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:17,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:17,961 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:17,961 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104866 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:17,961 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/B is initiating minor compaction (all files) 2023-04-03 13:59:17,961 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/A is initiating minor compaction (all files) 2023-04-03 13:59:17,961 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/B in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,961 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/A in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:17,961 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/afc57012d6a646b785789817e0f79616, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/9d455ddbc897438294e51ee71f6c4fa2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1a42f595f038415ba074d681a761605c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/64e3ed048b07427fbcb3cd214c2cca98] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=48.3 K 2023-04-03 13:59:17,961 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/4417ee30e156410d96ad469e514fe860, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/a9e5eecba9b64f52802846bf1e931d88, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/354067dcccfc4d7b8241e1559a09eb91, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/df660e18e79d46deac5f502a1bfcab5d] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=102.4 K 2023-04-03 13:59:17,961 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting afc57012d6a646b785789817e0f79616, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680530354219 2023-04-03 13:59:17,962 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 4417ee30e156410d96ad469e514fe860, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680530354219 2023-04-03 13:59:17,962 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 9d455ddbc897438294e51ee71f6c4fa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680530354366 2023-04-03 13:59:17,962 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting a9e5eecba9b64f52802846bf1e931d88, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680530354366 2023-04-03 13:59:17,962 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1a42f595f038415ba074d681a761605c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680530354484 2023-04-03 13:59:17,962 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting 354067dcccfc4d7b8241e1559a09eb91, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680530354484 2023-04-03 13:59:17,962 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 64e3ed048b07427fbcb3cd214c2cca98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680530356700 2023-04-03 13:59:17,962 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] compactions.Compactor(207): Compacting df660e18e79d46deac5f502a1bfcab5d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680530356700 2023-04-03 13:59:17,969 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#B#compaction#514 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:17,972 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#A#compaction#513 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:18,003 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:18,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:18,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-04-03 13:59:18,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 13:59:18,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 13:59:18,006 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 13:59:18,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:18,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 13:59:18,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 13:59:18,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:18,006 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:18,007 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:18,007 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:18,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:18,007 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:18,007 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 13:59:18,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:18,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 13:59:18,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:18,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 13:59:18,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 13:59:18,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 13:59:18,009 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 13:59:18,009 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 13:59:18,009 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 13:59:18,009 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 13:59:18,009 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 13:59:18,009 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:18,009 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. started... 2023-04-03 13:59:18,009 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 13:59:18,009 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:18,009 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:18,010 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:18,010 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:18,010 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:18,010 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:18,023 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403caa98e1afe2f47aaaec6e81b713fdf4e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403caa98e1afe2f47aaaec6e81b713fdf4e 2023-04-03 13:59:18,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:18,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. as already flushing 2023-04-03 13:59:18,278 DEBUG [Thread-3442] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bec0400 to 127.0.0.1:57081 2023-04-03 13:59:18,278 DEBUG [Thread-3442] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:18,279 DEBUG [Thread-3444] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34b8bf14 to 127.0.0.1:57081 2023-04-03 13:59:18,279 DEBUG [Thread-3444] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:18,280 DEBUG [Thread-3446] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x387217bb to 127.0.0.1:57081 2023-04-03 13:59:18,281 DEBUG [Thread-3446] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:18,282 DEBUG [Thread-3437] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f11e84a to 127.0.0.1:57081 2023-04-03 13:59:18,282 DEBUG [Thread-3437] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:18,283 DEBUG [Thread-3440] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ae3ec16 to 127.0.0.1:57081 2023-04-03 13:59:18,283 DEBUG [Thread-3440] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:18,290 DEBUG [Thread-3448] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c00a18c to 127.0.0.1:57081 2023-04-03 13:59:18,290 DEBUG [Thread-3448] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:18,385 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/76c2170fc1a04c0d9734c2845bcc5857 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/76c2170fc1a04c0d9734c2845bcc5857 2023-04-03 13:59:18,389 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/B of af4327b273d03b2199ab4cc1b2fa1539 into 76c2170fc1a04c0d9734c2845bcc5857(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:18,389 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:18,389 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/B, priority=12, startTime=1680530357960; duration=0sec 2023-04-03 13:59:18,389 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:18,389 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 13:59:18,390 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 13:59:18,390 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1912): af4327b273d03b2199ab4cc1b2fa1539/C is initiating minor compaction (all files) 2023-04-03 13:59:18,390 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2259): Starting compaction of af4327b273d03b2199ab4cc1b2fa1539/C in TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:18,390 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/90b0c0072a5a4e18b1e7caa5846e4ccc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1ba5bb55e4e343588bbfd04948b37256, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f5b095c8840142248b5318d855a63e64, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1e493784e4694f32920153aea84c39e9] into tmpdir=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp, totalSize=48.3 K 2023-04-03 13:59:18,390 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 90b0c0072a5a4e18b1e7caa5846e4ccc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680530354219 2023-04-03 13:59:18,391 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1ba5bb55e4e343588bbfd04948b37256, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680530354366 2023-04-03 13:59:18,391 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting f5b095c8840142248b5318d855a63e64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=470, earliestPutTs=1680530354484 2023-04-03 13:59:18,391 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] compactions.Compactor(207): Compacting 1e493784e4694f32920153aea84c39e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=508, earliestPutTs=1680530356700 2023-04-03 13:59:18,395 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] throttle.PressureAwareThroughputController(145): af4327b273d03b2199ab4cc1b2fa1539#C#compaction#516 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 13:59:18,428 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=517, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/945f65aa541648469a05cc6196e004b7 2023-04-03 13:59:18,667 DEBUG [Thread-3433] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5c51ec to 127.0.0.1:57081 2023-04-03 13:59:18,667 DEBUG [Thread-3433] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:18,796 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/2d1873d00e68431aa86f7df0af5b7c11 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/2d1873d00e68431aa86f7df0af5b7c11 2023-04-03 13:59:18,802 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/A of af4327b273d03b2199ab4cc1b2fa1539 into 2d1873d00e68431aa86f7df0af5b7c11(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:18,802 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:18,802 INFO [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/A, priority=12, startTime=1680530357959; duration=0sec 2023-04-03 13:59:18,802 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:18,804 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/70daaa673e674873b718b6ac6ec72f2b as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/70daaa673e674873b718b6ac6ec72f2b 2023-04-03 13:59:18,808 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in af4327b273d03b2199ab4cc1b2fa1539/C of af4327b273d03b2199ab4cc1b2fa1539 into 70daaa673e674873b718b6ac6ec72f2b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 13:59:18,808 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:18,808 INFO [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539., storeName=af4327b273d03b2199ab4cc1b2fa1539/C, priority=12, startTime=1680530357960; duration=0sec 2023-04-03 13:59:18,808 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 13:59:18,825 DEBUG [Thread-3429] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58f3df8a to 127.0.0.1:57081 2023-04-03 13:59:18,825 DEBUG [Thread-3429] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:18,845 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/e1be9a87740846fb958f52747cba11ee 2023-04-03 13:59:19,265 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/e5c68fddf596405da5c2731d10777872 2023-04-03 13:59:19,273 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/945f65aa541648469a05cc6196e004b7 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/945f65aa541648469a05cc6196e004b7 2023-04-03 13:59:19,278 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/945f65aa541648469a05cc6196e004b7, entries=100, sequenceid=517, filesize=18.6 K 2023-04-03 13:59:19,279 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/e1be9a87740846fb958f52747cba11ee as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e1be9a87740846fb958f52747cba11ee 2023-04-03 13:59:19,284 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e1be9a87740846fb958f52747cba11ee, entries=100, sequenceid=517, filesize=9.5 K 2023-04-03 13:59:19,284 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/e5c68fddf596405da5c2731d10777872 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e5c68fddf596405da5c2731d10777872 2023-04-03 13:59:19,289 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e5c68fddf596405da5c2731d10777872, entries=100, sequenceid=517, filesize=9.5 K 2023-04-03 13:59:19,290 INFO [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=40.25 KB/41220 for af4327b273d03b2199ab4cc1b2fa1539 in 1281ms, sequenceid=517, compaction requested=false 2023-04-03 13:59:19,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:19,290 DEBUG [rs(jenkins-hbase20.apache.org,39695,1680530174856)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:19,290 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 13:59:19,290 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 13:59:19,290 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 13:59:19,291 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 13:59:19,292 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:19,292 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:19,292 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:19,292 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 13:59:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:19,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,39695,1680530174856' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 13:59:19,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d46533[Count = 0] remaining members to acquire global barrier 2023-04-03 13:59:19,295 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 13:59:19,295 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:19,295 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:19,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:19,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:19,296 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 13:59:19,296 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 13:59:19,296 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,39695,1680530174856' in zk 2023-04-03 13:59:19,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,296 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 13:59:19,297 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,297 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 13:59:19,297 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:19,297 DEBUG [member: 'jenkins-hbase20.apache.org,39695,1680530174856' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 13:59:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,39695,1680530174856': 2023-04-03 13:59:19,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,39695,1680530174856' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 13:59:19,301 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 13:59:19,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 13:59:19,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 13:59:19,301 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 13:59:19,301 INFO [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 13:59:19,308 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:19,308 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:19,309 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:19,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 13:59:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 13:59:19,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:19,309 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 13:59:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 13:59:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:19,312 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 13:59:19,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 13:59:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 13:59:19,317 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:19,318 DEBUG [(jenkins-hbase20.apache.org,37337,1680530173952)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 13:59:19,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 13:59:19,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 13:59:19,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 13:59:19,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:19,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 13:59:19,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 13:59:19,318 DEBUG [Thread-3439] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 13:59:19,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 13:59:19,318 DEBUG [Thread-3439] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 13:59:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 13:59:19,318 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 13:59:19,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 13:59:19,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 13:59:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 13:59:19,418 DEBUG [Thread-3439] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 13:59:19,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 13:59:19,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39695] regionserver.HRegion(9158): Flush requested on af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:19,682 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 13:59:19,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:19,683 DEBUG [Thread-3435] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5be55982 to 127.0.0.1:57081 2023-04-03 13:59:19,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:19,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:19,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:19,683 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:19,683 DEBUG [Thread-3435] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:19,683 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:19,699 DEBUG [Thread-3431] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c217430 to 127.0.0.1:57081 2023-04-03 13:59:19,699 DEBUG [Thread-3431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:19,699 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 13:59:19,699 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 165 2023-04-03 13:59:19,700 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-03 13:59:19,700 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-04-03 13:59:19,700 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-04-03 13:59:19,700 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-04-03 13:59:19,700 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 13:59:19,700 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5080 2023-04-03 13:59:19,700 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5224 2023-04-03 13:59:19,700 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5021 2023-04-03 13:59:19,701 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5099 2023-04-03 13:59:19,701 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(398): read 5233 2023-04-03 13:59:19,701 INFO [Listener at localhost.localdomain/43477] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 13:59:19,701 INFO [Listener at localhost.localdomain/43477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 13:59:19,701 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77459ab3 to 127.0.0.1:57081 2023-04-03 13:59:19,701 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:19,702 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 13:59:19,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-04-03 13:59:19,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 13:59:19,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 13:59:19,706 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530359706"}]},"ts":"1680530359706"} 2023-04-03 13:59:19,707 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 13:59:19,708 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 13:59:19,708 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, UNASSIGN}] 2023-04-03 13:59:19,709 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, UNASSIGN 2023-04-03 13:59:19,710 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=af4327b273d03b2199ab4cc1b2fa1539, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:19,710 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530359710"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1680530359710"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680530359710"}]},"ts":"1680530359710"} 2023-04-03 13:59:19,711 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856}] 2023-04-03 13:59:19,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 13:59:19,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:19,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing af4327b273d03b2199ab4cc1b2fa1539, disabling compactions & flushes 2023-04-03 13:59:19,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:20,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 13:59:20,102 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403ef08b40bfb294aa8ad0a68b161205263 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ef08b40bfb294aa8ad0a68b161205263 2023-04-03 13:59:20,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 13:59:20,508 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=530, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/11c62aa835de463b949ac3e4d72b1c79 2023-04-03 13:59:20,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/57a5bba835ae4545a140ce2cb9d6f21d 2023-04-03 13:59:20,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/929a67ac1be24fe2b8af1fd4898d2f60 2023-04-03 13:59:20,540 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/11c62aa835de463b949ac3e4d72b1c79 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/11c62aa835de463b949ac3e4d72b1c79 2023-04-03 13:59:20,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/11c62aa835de463b949ac3e4d72b1c79, entries=100, sequenceid=530, filesize=18.6 K 2023-04-03 13:59:20,543 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/57a5bba835ae4545a140ce2cb9d6f21d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/57a5bba835ae4545a140ce2cb9d6f21d 2023-04-03 13:59:20,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/57a5bba835ae4545a140ce2cb9d6f21d, entries=100, sequenceid=530, filesize=9.5 K 2023-04-03 13:59:20,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/929a67ac1be24fe2b8af1fd4898d2f60 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/929a67ac1be24fe2b8af1fd4898d2f60 2023-04-03 13:59:20,550 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/929a67ac1be24fe2b8af1fd4898d2f60, entries=100, sequenceid=530, filesize=9.5 K 2023-04-03 13:59:20,550 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=6.71 KB/6870 for af4327b273d03b2199ab4cc1b2fa1539 in 868ms, sequenceid=530, compaction requested=true 2023-04-03 13:59:20,551 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:20,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:20,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:20,551 DEBUG [RS:0;jenkins-hbase20:39695-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. because compaction request was cancelled 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. after waiting 0 ms 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:20,551 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. because compaction request was cancelled 2023-04-03 13:59:20,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:20,551 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing af4327b273d03b2199ab4cc1b2fa1539 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-03 13:59:20,551 DEBUG [RS:0;jenkins-hbase20:39695-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. because compaction request was cancelled 2023-04-03 13:59:20,551 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=A 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=B 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK af4327b273d03b2199ab4cc1b2fa1539, store=C 2023-04-03 13:59:20,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 13:59:20,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 13:59:20,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403f5ba7626041f44b39d151837154de25c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f5ba7626041f44b39d151837154de25c 2023-04-03 13:59:21,375 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=534, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/cc1c6c3270284d3091afd4c3441b4066 2023-04-03 13:59:21,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/79271e063636484c8fec85e126296e76 2023-04-03 13:59:21,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/d186d86396584c488151071d2f670c4d 2023-04-03 13:59:21,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/A/cc1c6c3270284d3091afd4c3441b4066 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/cc1c6c3270284d3091afd4c3441b4066 2023-04-03 13:59:21,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/cc1c6c3270284d3091afd4c3441b4066, entries=50, sequenceid=534, filesize=11.7 K 2023-04-03 13:59:21,813 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/B/79271e063636484c8fec85e126296e76 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/79271e063636484c8fec85e126296e76 2023-04-03 13:59:21,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 13:59:21,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/79271e063636484c8fec85e126296e76, entries=50, sequenceid=534, filesize=7.1 K 2023-04-03 13:59:21,820 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/.tmp/C/d186d86396584c488151071d2f670c4d as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d186d86396584c488151071d2f670c4d 2023-04-03 13:59:21,826 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d186d86396584c488151071d2f670c4d, entries=50, sequenceid=534, filesize=7.1 K 2023-04-03 13:59:21,827 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=0 B/0 for af4327b273d03b2199ab4cc1b2fa1539 in 1276ms, sequenceid=534, compaction requested=true 2023-04-03 13:59:21,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/6b6a4786de414718bd3dd1b487719b97, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c00c049e7ab542d89b8e3c002a7647a3, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/59c74cce6ac54bce9177c25f9644e774, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/92d1f2315f1543c88cdee95d51bae49f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa9db53527b14c588d1f9e296d8ae2c9, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c0f91c0312f04ae6a002a0a468f2c151, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/93b854ec6e674fdd8400f993de3318b1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d1299f452cbe4571852af221ddb5c689, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa23d20d85434d2eac5d028b314a64c0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/ca2a3a24f68a4116a7e1489be6c190cb, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/dc9de894e9ea45f4a7c29c4ad08c7ab8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/b3b506df1b154a4d93fd6a141442a3f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/339ca567b9b44b9ca86516570f61874a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/31440f7c3c254b038c1907561d88ce7b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/66aef0071d3f490093f967b1b1619f67, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d857bec3b3b64e95a7e7577a7cbf53f5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c6534b50d6bf4c668911ee36dcee31ef, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/0982582878724f23950726eafd2905c5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/eb4e9125f4bc481ea8349bc17a30ae9d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa8d5fed29c645c3b05e0cf853de6a9c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/4417ee30e156410d96ad469e514fe860, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/a9e5eecba9b64f52802846bf1e931d88, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/354067dcccfc4d7b8241e1559a09eb91, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/df660e18e79d46deac5f502a1bfcab5d] to archive 2023-04-03 13:59:21,830 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:59:21,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/6b6a4786de414718bd3dd1b487719b97 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/6b6a4786de414718bd3dd1b487719b97 2023-04-03 13:59:21,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c00c049e7ab542d89b8e3c002a7647a3 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c00c049e7ab542d89b8e3c002a7647a3 2023-04-03 13:59:21,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/59c74cce6ac54bce9177c25f9644e774 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/59c74cce6ac54bce9177c25f9644e774 2023-04-03 13:59:21,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/92d1f2315f1543c88cdee95d51bae49f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/92d1f2315f1543c88cdee95d51bae49f 2023-04-03 13:59:21,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa9db53527b14c588d1f9e296d8ae2c9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa9db53527b14c588d1f9e296d8ae2c9 2023-04-03 13:59:21,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c0f91c0312f04ae6a002a0a468f2c151 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c0f91c0312f04ae6a002a0a468f2c151 2023-04-03 13:59:21,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/93b854ec6e674fdd8400f993de3318b1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/93b854ec6e674fdd8400f993de3318b1 2023-04-03 13:59:21,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d1299f452cbe4571852af221ddb5c689 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d1299f452cbe4571852af221ddb5c689 2023-04-03 13:59:21,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa23d20d85434d2eac5d028b314a64c0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa23d20d85434d2eac5d028b314a64c0 2023-04-03 13:59:21,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/ca2a3a24f68a4116a7e1489be6c190cb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/ca2a3a24f68a4116a7e1489be6c190cb 2023-04-03 13:59:21,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/dc9de894e9ea45f4a7c29c4ad08c7ab8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/dc9de894e9ea45f4a7c29c4ad08c7ab8 2023-04-03 13:59:21,843 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/b3b506df1b154a4d93fd6a141442a3f5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/b3b506df1b154a4d93fd6a141442a3f5 2023-04-03 13:59:21,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/339ca567b9b44b9ca86516570f61874a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/339ca567b9b44b9ca86516570f61874a 2023-04-03 13:59:21,845 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/31440f7c3c254b038c1907561d88ce7b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/31440f7c3c254b038c1907561d88ce7b 2023-04-03 13:59:21,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/66aef0071d3f490093f967b1b1619f67 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/66aef0071d3f490093f967b1b1619f67 2023-04-03 13:59:21,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d857bec3b3b64e95a7e7577a7cbf53f5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/d857bec3b3b64e95a7e7577a7cbf53f5 2023-04-03 13:59:21,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c6534b50d6bf4c668911ee36dcee31ef to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/c6534b50d6bf4c668911ee36dcee31ef 2023-04-03 13:59:21,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/0982582878724f23950726eafd2905c5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/0982582878724f23950726eafd2905c5 2023-04-03 13:59:21,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/eb4e9125f4bc481ea8349bc17a30ae9d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/eb4e9125f4bc481ea8349bc17a30ae9d 2023-04-03 13:59:21,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa8d5fed29c645c3b05e0cf853de6a9c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/aa8d5fed29c645c3b05e0cf853de6a9c 2023-04-03 13:59:21,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/4417ee30e156410d96ad469e514fe860 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/4417ee30e156410d96ad469e514fe860 2023-04-03 13:59:21,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/a9e5eecba9b64f52802846bf1e931d88 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/a9e5eecba9b64f52802846bf1e931d88 2023-04-03 13:59:21,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/354067dcccfc4d7b8241e1559a09eb91 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/354067dcccfc4d7b8241e1559a09eb91 2023-04-03 13:59:21,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/df660e18e79d46deac5f502a1bfcab5d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/df660e18e79d46deac5f502a1bfcab5d 2023-04-03 13:59:21,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11bb841c96be4eb0863e539c0e55993f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/b4710114a36141ae9d465cb9b3cd5df6, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/016b4ae1037d4035be0b568899ae65a8, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/86dc8ae0491d4a3c9dff75432c4f9762, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bc16d9d7a24843a28efc9b89ec06d976, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d99472743cd04bd4aad954388ceb2640, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2e0ef5f1f0ef4f6f9dec43316f2718bf, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/293a2e65aafb4e768563c52a55a50a94, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11ab2c30f3a546c096344d239b900c2f, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/003f26533cd74e699a91b07f2417e10c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/52827c2b03fb49b4bba3fc4c23ade9ad, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2fb9248c8f054c6db7e3d39c52aa4f0a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/eff8d7de2b42451ca5c78a81e675a399, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/fb90101d87384395909a254e27ccd9b4, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bee0aee47e2d4b34aece936fa1d8e5de, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1305b9eea8c24754ae49fef2971aaf34, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/c09720af9cc549c0a55a8b06d16d8b73, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e3e2b94ce885417fb24caa5b50e9757d, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/075f034f93ac47d6b595e1c0d18abd72, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/afc57012d6a646b785789817e0f79616, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d7ae33a847df4c5aae38ad90dd58ee30, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/9d455ddbc897438294e51ee71f6c4fa2, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1a42f595f038415ba074d681a761605c, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/64e3ed048b07427fbcb3cd214c2cca98] to archive 2023-04-03 13:59:21,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:59:21,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11bb841c96be4eb0863e539c0e55993f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11bb841c96be4eb0863e539c0e55993f 2023-04-03 13:59:21,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/b4710114a36141ae9d465cb9b3cd5df6 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/b4710114a36141ae9d465cb9b3cd5df6 2023-04-03 13:59:21,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/016b4ae1037d4035be0b568899ae65a8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/016b4ae1037d4035be0b568899ae65a8 2023-04-03 13:59:21,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/86dc8ae0491d4a3c9dff75432c4f9762 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/86dc8ae0491d4a3c9dff75432c4f9762 2023-04-03 13:59:21,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bc16d9d7a24843a28efc9b89ec06d976 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bc16d9d7a24843a28efc9b89ec06d976 2023-04-03 13:59:21,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d99472743cd04bd4aad954388ceb2640 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d99472743cd04bd4aad954388ceb2640 2023-04-03 13:59:21,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2e0ef5f1f0ef4f6f9dec43316f2718bf to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2e0ef5f1f0ef4f6f9dec43316f2718bf 2023-04-03 13:59:21,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/293a2e65aafb4e768563c52a55a50a94 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/293a2e65aafb4e768563c52a55a50a94 2023-04-03 13:59:21,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11ab2c30f3a546c096344d239b900c2f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/11ab2c30f3a546c096344d239b900c2f 2023-04-03 13:59:21,862 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/003f26533cd74e699a91b07f2417e10c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/003f26533cd74e699a91b07f2417e10c 2023-04-03 13:59:21,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/52827c2b03fb49b4bba3fc4c23ade9ad to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/52827c2b03fb49b4bba3fc4c23ade9ad 2023-04-03 13:59:21,864 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2fb9248c8f054c6db7e3d39c52aa4f0a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/2fb9248c8f054c6db7e3d39c52aa4f0a 2023-04-03 13:59:21,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/eff8d7de2b42451ca5c78a81e675a399 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/eff8d7de2b42451ca5c78a81e675a399 2023-04-03 13:59:21,865 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/fb90101d87384395909a254e27ccd9b4 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/fb90101d87384395909a254e27ccd9b4 2023-04-03 13:59:21,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bee0aee47e2d4b34aece936fa1d8e5de to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/bee0aee47e2d4b34aece936fa1d8e5de 2023-04-03 13:59:21,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1305b9eea8c24754ae49fef2971aaf34 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1305b9eea8c24754ae49fef2971aaf34 2023-04-03 13:59:21,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/c09720af9cc549c0a55a8b06d16d8b73 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/c09720af9cc549c0a55a8b06d16d8b73 2023-04-03 13:59:21,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e3e2b94ce885417fb24caa5b50e9757d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e3e2b94ce885417fb24caa5b50e9757d 2023-04-03 13:59:21,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/075f034f93ac47d6b595e1c0d18abd72 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/075f034f93ac47d6b595e1c0d18abd72 2023-04-03 13:59:21,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/afc57012d6a646b785789817e0f79616 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/afc57012d6a646b785789817e0f79616 2023-04-03 13:59:21,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d7ae33a847df4c5aae38ad90dd58ee30 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/d7ae33a847df4c5aae38ad90dd58ee30 2023-04-03 13:59:21,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/9d455ddbc897438294e51ee71f6c4fa2 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/9d455ddbc897438294e51ee71f6c4fa2 2023-04-03 13:59:21,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1a42f595f038415ba074d681a761605c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/1a42f595f038415ba074d681a761605c 2023-04-03 13:59:21,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/64e3ed048b07427fbcb3cd214c2cca98 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/64e3ed048b07427fbcb3cd214c2cca98 2023-04-03 13:59:21,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/0cf805cacdee4219947519395b77c564, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/77f69a46fe454d638c9f029cf90fe3a5, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7de0821458f34b049737f92aabc9ae45, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/6e05c900670e4cbaa0de163a36a38a0b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/9702d717df714b138099e95f653cf046, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/86fe47bd118448c89e67bc6f29939eb1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/308492eba68f4681ad510b69c6f0e40b, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7d45d23797b445ef9f2ffff5932d8006, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d37c77462b494696a8a97cf08d95073a, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2770ca780f114f26837155a7b4099964, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d2d48e29fceb45908cb20bbb211d31f0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e07232af82e94c58950812efe5de7ed1, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2054a29bc9be4096a05b520025788a97, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/bc11fbef6db744409607cb0959819749, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/4d25f17f11e84fe39a0be1d3dcdc4269, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ed50dcb195164c6da8e6b83b22b53941, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ec28eb8a88124ea292989d80bb594508, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f409162096254a9cbf1177f924fa2a17, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/c73e7cd620f34433b1ae4c6fbef896d0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/90b0c0072a5a4e18b1e7caa5846e4ccc, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/89c608c193c14451a9666be72ef3c2a0, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1ba5bb55e4e343588bbfd04948b37256, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f5b095c8840142248b5318d855a63e64, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1e493784e4694f32920153aea84c39e9] to archive 2023-04-03 13:59:21,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 13:59:21,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/0cf805cacdee4219947519395b77c564 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/0cf805cacdee4219947519395b77c564 2023-04-03 13:59:21,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/77f69a46fe454d638c9f029cf90fe3a5 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/77f69a46fe454d638c9f029cf90fe3a5 2023-04-03 13:59:21,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7de0821458f34b049737f92aabc9ae45 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7de0821458f34b049737f92aabc9ae45 2023-04-03 13:59:21,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/6e05c900670e4cbaa0de163a36a38a0b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/6e05c900670e4cbaa0de163a36a38a0b 2023-04-03 13:59:21,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/9702d717df714b138099e95f653cf046 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/9702d717df714b138099e95f653cf046 2023-04-03 13:59:21,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/86fe47bd118448c89e67bc6f29939eb1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/86fe47bd118448c89e67bc6f29939eb1 2023-04-03 13:59:21,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/308492eba68f4681ad510b69c6f0e40b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/308492eba68f4681ad510b69c6f0e40b 2023-04-03 13:59:21,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7d45d23797b445ef9f2ffff5932d8006 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/7d45d23797b445ef9f2ffff5932d8006 2023-04-03 13:59:21,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d37c77462b494696a8a97cf08d95073a to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d37c77462b494696a8a97cf08d95073a 2023-04-03 13:59:21,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2770ca780f114f26837155a7b4099964 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2770ca780f114f26837155a7b4099964 2023-04-03 13:59:21,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d2d48e29fceb45908cb20bbb211d31f0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d2d48e29fceb45908cb20bbb211d31f0 2023-04-03 13:59:21,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e07232af82e94c58950812efe5de7ed1 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e07232af82e94c58950812efe5de7ed1 2023-04-03 13:59:21,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2054a29bc9be4096a05b520025788a97 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/2054a29bc9be4096a05b520025788a97 2023-04-03 13:59:21,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/bc11fbef6db744409607cb0959819749 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/bc11fbef6db744409607cb0959819749 2023-04-03 13:59:21,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/4d25f17f11e84fe39a0be1d3dcdc4269 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/4d25f17f11e84fe39a0be1d3dcdc4269 2023-04-03 13:59:21,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ed50dcb195164c6da8e6b83b22b53941 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ed50dcb195164c6da8e6b83b22b53941 2023-04-03 13:59:21,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ec28eb8a88124ea292989d80bb594508 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/ec28eb8a88124ea292989d80bb594508 2023-04-03 13:59:21,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f409162096254a9cbf1177f924fa2a17 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f409162096254a9cbf1177f924fa2a17 2023-04-03 13:59:21,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/c73e7cd620f34433b1ae4c6fbef896d0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/c73e7cd620f34433b1ae4c6fbef896d0 2023-04-03 13:59:21,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/90b0c0072a5a4e18b1e7caa5846e4ccc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/90b0c0072a5a4e18b1e7caa5846e4ccc 2023-04-03 13:59:21,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/89c608c193c14451a9666be72ef3c2a0 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/89c608c193c14451a9666be72ef3c2a0 2023-04-03 13:59:21,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1ba5bb55e4e343588bbfd04948b37256 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1ba5bb55e4e343588bbfd04948b37256 2023-04-03 13:59:21,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f5b095c8840142248b5318d855a63e64 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/f5b095c8840142248b5318d855a63e64 2023-04-03 13:59:21,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1e493784e4694f32920153aea84c39e9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/1e493784e4694f32920153aea84c39e9 2023-04-03 13:59:21,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/recovered.edits/537.seqid, newMaxSeqId=537, maxSeqId=4 2023-04-03 13:59:21,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539. 2023-04-03 13:59:21,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for af4327b273d03b2199ab4cc1b2fa1539: 2023-04-03 13:59:21,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:21,896 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=af4327b273d03b2199ab4cc1b2fa1539, regionState=CLOSED 2023-04-03 13:59:21,896 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680530361896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680530361896"}]},"ts":"1680530361896"} 2023-04-03 13:59:21,898 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-03 13:59:21,898 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure af4327b273d03b2199ab4cc1b2fa1539, server=jenkins-hbase20.apache.org,39695,1680530174856 in 2.1860 sec 2023-04-03 13:59:21,899 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-03 13:59:21,899 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=af4327b273d03b2199ab4cc1b2fa1539, UNASSIGN in 2.1900 sec 2023-04-03 13:59:21,899 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680530361899"}]},"ts":"1680530361899"} 2023-04-03 13:59:21,900 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 13:59:22,030 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 13:59:22,034 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.3280 sec 2023-04-03 13:59:23,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 13:59:23,816 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-03 13:59:23,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-04-03 13:59:23,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:59:23,818 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:59:23,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-03 13:59:23,819 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:59:23,823 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:23,825 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C, FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/recovered.edits] 2023-04-03 13:59:23,828 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/11c62aa835de463b949ac3e4d72b1c79 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/11c62aa835de463b949ac3e4d72b1c79 2023-04-03 13:59:23,829 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/2d1873d00e68431aa86f7df0af5b7c11 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/2d1873d00e68431aa86f7df0af5b7c11 2023-04-03 13:59:23,830 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/945f65aa541648469a05cc6196e004b7 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/945f65aa541648469a05cc6196e004b7 2023-04-03 13:59:23,831 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/cc1c6c3270284d3091afd4c3441b4066 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/A/cc1c6c3270284d3091afd4c3441b4066 2023-04-03 13:59:23,833 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/57a5bba835ae4545a140ce2cb9d6f21d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/57a5bba835ae4545a140ce2cb9d6f21d 2023-04-03 13:59:23,834 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/76c2170fc1a04c0d9734c2845bcc5857 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/76c2170fc1a04c0d9734c2845bcc5857 2023-04-03 13:59:23,834 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/79271e063636484c8fec85e126296e76 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/79271e063636484c8fec85e126296e76 2023-04-03 13:59:23,836 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e1be9a87740846fb958f52747cba11ee to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/B/e1be9a87740846fb958f52747cba11ee 2023-04-03 13:59:23,838 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/70daaa673e674873b718b6ac6ec72f2b to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/70daaa673e674873b718b6ac6ec72f2b 2023-04-03 13:59:23,839 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/929a67ac1be24fe2b8af1fd4898d2f60 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/929a67ac1be24fe2b8af1fd4898d2f60 2023-04-03 13:59:23,840 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d186d86396584c488151071d2f670c4d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/d186d86396584c488151071d2f670c4d 2023-04-03 13:59:23,841 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e5c68fddf596405da5c2731d10777872 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/C/e5c68fddf596405da5c2731d10777872 2023-04-03 13:59:23,843 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/recovered.edits/537.seqid to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539/recovered.edits/537.seqid 2023-04-03 13:59:23,843 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/.tmp/data/default/TestAcidGuarantees/af4327b273d03b2199ab4cc1b2fa1539 2023-04-03 13:59:23,843 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 13:59:23,843 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 13:59:23,844 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 13:59:23,846 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403061e0ba05ee446f095c41ce5e2c50e42 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403061e0ba05ee446f095c41ce5e2c50e42 2023-04-03 13:59:23,847 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030bff1940450944828fba1e4192c2666f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030bff1940450944828fba1e4192c2666f 2023-04-03 13:59:23,848 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030c41e4201842489abd87db95cec71d07 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030c41e4201842489abd87db95cec71d07 2023-04-03 13:59:23,849 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040314de667e1df8407688285f65a45d4902 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040314de667e1df8407688285f65a45d4902 2023-04-03 13:59:23,850 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040338dcdcf535ff4197b69f89c07262a61d to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040338dcdcf535ff4197b69f89c07262a61d 2023-04-03 13:59:23,851 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034a23b21895964ff789abc6fac47cd6cb to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304034a23b21895964ff789abc6fac47cd6cb 2023-04-03 13:59:23,852 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403510034bdd30e46fca69987c400fcda59 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403510034bdd30e46fca69987c400fcda59 2023-04-03 13:59:23,853 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035feb17a252474ff59c4a74bf6e6af58c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035feb17a252474ff59c4a74bf6e6af58c 2023-04-03 13:59:23,854 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036959ba73039441878d62de8a805f8691 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036959ba73039441878d62de8a805f8691 2023-04-03 13:59:23,855 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403733a6d7bec6246548ebb9fb66d7003c9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403733a6d7bec6246548ebb9fb66d7003c9 2023-04-03 13:59:23,856 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037618c126d67d425e903bb0f10fc26a76 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037618c126d67d425e903bb0f10fc26a76 2023-04-03 13:59:23,857 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040385df8f1ca17a42b8a55fefc035a6a6e8 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040385df8f1ca17a42b8a55fefc035a6a6e8 2023-04-03 13:59:23,857 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038649727dbe43485a9d4d0a695d6227cc to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038649727dbe43485a9d4d0a695d6227cc 2023-04-03 13:59:23,859 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038d8121d45e3b41cab47b0d31ec44ff6e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038d8121d45e3b41cab47b0d31ec44ff6e 2023-04-03 13:59:23,860 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038df1eaaad24d43568c6015d196d8b383 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038df1eaaad24d43568c6015d196d8b383 2023-04-03 13:59:23,861 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038e0a5da400d5412fa340a8820aa186ef to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038e0a5da400d5412fa340a8820aa186ef 2023-04-03 13:59:23,863 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039e7042f365d14d3f9b63d2a069a08d2f to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304039e7042f365d14d3f9b63d2a069a08d2f 2023-04-03 13:59:23,864 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a87f43214a7843129e8caf8a58e29936 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a87f43214a7843129e8caf8a58e29936 2023-04-03 13:59:23,865 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c709c0e0ce3b467daff60395065c0bb9 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c709c0e0ce3b467daff60395065c0bb9 2023-04-03 13:59:23,866 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403caa98e1afe2f47aaaec6e81b713fdf4e to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403caa98e1afe2f47aaaec6e81b713fdf4e 2023-04-03 13:59:23,868 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e49475e982ee4c7e83fe78bdc87d8816 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e49475e982ee4c7e83fe78bdc87d8816 2023-04-03 13:59:23,869 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ef08b40bfb294aa8ad0a68b161205263 to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ef08b40bfb294aa8ad0a68b161205263 2023-04-03 13:59:23,870 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f5ba7626041f44b39d151837154de25c to hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f5ba7626041f44b39d151837154de25c 2023-04-03 13:59:23,870 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 13:59:23,872 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:59:23,873 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 13:59:23,874 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 13:59:23,875 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:59:23,875 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 13:59:23,875 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680530363875"}]},"ts":"9223372036854775807"} 2023-04-03 13:59:23,876 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 13:59:23,876 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => af4327b273d03b2199ab4cc1b2fa1539, NAME => 'TestAcidGuarantees,,1680530336119.af4327b273d03b2199ab4cc1b2fa1539.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 13:59:23,876 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 13:59:23,876 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680530363876"}]},"ts":"9223372036854775807"} 2023-04-03 13:59:23,877 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 13:59:23,878 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 13:59:23,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 62 msec 2023-04-03 13:59:23,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37337] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-03 13:59:23,920 INFO [Listener at localhost.localdomain/43477] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-03 13:59:23,936 INFO [Listener at localhost.localdomain/43477] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 225) - Thread LEAK? -, OpenFileDescriptor=557 (was 520) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=299 (was 307), ProcessCount=166 (was 166), AvailableMemoryMB=3697 (was 3580) - AvailableMemoryMB LEAK? - 2023-04-03 13:59:23,937 INFO [Listener at localhost.localdomain/43477] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-03 13:59:23,937 INFO [Listener at localhost.localdomain/43477] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 13:59:23,937 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d5f4139 to 127.0.0.1:57081 2023-04-03 13:59:23,937 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:23,937 DEBUG [Listener at localhost.localdomain/43477] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-03 13:59:23,937 DEBUG [Listener at localhost.localdomain/43477] util.JVMClusterUtil(257): Found active master hash=866943624, stopped=false 2023-04-03 13:59:23,938 INFO [Listener at localhost.localdomain/43477] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:59:23,938 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-03 13:59:23,938 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-03 13:59:23,938 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:59:23,938 INFO [Listener at localhost.localdomain/43477] procedure2.ProcedureExecutor(629): Stopping 2023-04-03 13:59:23,939 DEBUG [Listener at localhost.localdomain/43477] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07ff1af1 to 127.0.0.1:57081 2023-04-03 13:59:23,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 13:59:23,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 13:59:23,939 DEBUG [Listener at localhost.localdomain/43477] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:23,939 INFO [Listener at localhost.localdomain/43477] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,39695,1680530174856' ***** 2023-04-03 13:59:23,939 INFO [Listener at localhost.localdomain/43477] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-03 13:59:23,939 INFO [RS:0;jenkins-hbase20:39695] regionserver.HeapMemoryManager(220): Stopping 2023-04-03 13:59:23,940 INFO [RS:0;jenkins-hbase20:39695] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-03 13:59:23,940 INFO [RS:0;jenkins-hbase20:39695] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-03 13:59:23,940 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-03 13:59:23,940 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(3304): Received CLOSE for 1162c80dd821d6d790c48c84de83a7bd 2023-04-03 13:59:23,940 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:23,940 DEBUG [RS:0;jenkins-hbase20:39695] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d1034ff to 127.0.0.1:57081 2023-04-03 13:59:23,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1162c80dd821d6d790c48c84de83a7bd, disabling compactions & flushes 2023-04-03 13:59:23,941 DEBUG [RS:0;jenkins-hbase20:39695] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:23,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:59:23,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:59:23,941 INFO [RS:0;jenkins-hbase20:39695] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-03 13:59:23,941 INFO [RS:0;jenkins-hbase20:39695] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-03 13:59:23,941 INFO [RS:0;jenkins-hbase20:39695] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-03 13:59:23,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. after waiting 0 ms 2023-04-03 13:59:23,941 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-03 13:59:23,941 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:59:23,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1162c80dd821d6d790c48c84de83a7bd 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-03 13:59:23,941 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-03 13:59:23,941 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 1162c80dd821d6d790c48c84de83a7bd=hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd.} 2023-04-03 13:59:23,942 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-03 13:59:23,942 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-03 13:59:23,942 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-03 13:59:23,942 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-03 13:59:23,942 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-03 13:59:23,942 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-04-03 13:59:23,943 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1505): Waiting on 1162c80dd821d6d790c48c84de83a7bd, 1588230740 2023-04-03 13:59:24,031 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-03 13:59:24,144 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1505): Waiting on 1162c80dd821d6d790c48c84de83a7bd, 1588230740 2023-04-03 13:59:24,344 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1505): Waiting on 1162c80dd821d6d790c48c84de83a7bd, 1588230740 2023-04-03 13:59:24,348 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd/.tmp/info/d89455b3fca041e185056a6f09d4b347 2023-04-03 13:59:24,348 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/.tmp/info/7a131491c0364baf92bb38758e8ed9de 2023-04-03 13:59:24,349 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-03 13:59:24,349 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-03 13:59:24,357 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7a131491c0364baf92bb38758e8ed9de 2023-04-03 13:59:24,357 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd/.tmp/info/d89455b3fca041e185056a6f09d4b347 as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd/info/d89455b3fca041e185056a6f09d4b347 2023-04-03 13:59:24,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd/info/d89455b3fca041e185056a6f09d4b347, entries=2, sequenceid=6, filesize=4.8 K 2023-04-03 13:59:24,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 1162c80dd821d6d790c48c84de83a7bd in 421ms, sequenceid=6, compaction requested=false 2023-04-03 13:59:24,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/namespace/1162c80dd821d6d790c48c84de83a7bd/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-03 13:59:24,370 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:59:24,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1162c80dd821d6d790c48c84de83a7bd: 2023-04-03 13:59:24,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680530177268.1162c80dd821d6d790c48c84de83a7bd. 2023-04-03 13:59:24,544 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 13:59:24,745 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 13:59:24,768 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/.tmp/rep_barrier/2dafba5fc5be41bbae12e735b08e243e 2023-04-03 13:59:24,772 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2dafba5fc5be41bbae12e735b08e243e 2023-04-03 13:59:24,945 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-03 13:59:24,945 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-03 13:59:24,945 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 13:59:25,146 DEBUG [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 13:59:25,179 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/.tmp/table/48ec9cd63e1540cbb6bb4250f5f418be 2023-04-03 13:59:25,187 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 48ec9cd63e1540cbb6bb4250f5f418be 2023-04-03 13:59:25,188 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/.tmp/info/7a131491c0364baf92bb38758e8ed9de as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/info/7a131491c0364baf92bb38758e8ed9de 2023-04-03 13:59:25,196 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 7a131491c0364baf92bb38758e8ed9de 2023-04-03 13:59:25,196 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/info/7a131491c0364baf92bb38758e8ed9de, entries=22, sequenceid=93, filesize=7.1 K 2023-04-03 13:59:25,198 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/.tmp/rep_barrier/2dafba5fc5be41bbae12e735b08e243e as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/rep_barrier/2dafba5fc5be41bbae12e735b08e243e 2023-04-03 13:59:25,203 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2dafba5fc5be41bbae12e735b08e243e 2023-04-03 13:59:25,203 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/rep_barrier/2dafba5fc5be41bbae12e735b08e243e, entries=6, sequenceid=93, filesize=5.5 K 2023-04-03 13:59:25,204 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/.tmp/table/48ec9cd63e1540cbb6bb4250f5f418be as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/table/48ec9cd63e1540cbb6bb4250f5f418be 2023-04-03 13:59:25,209 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 48ec9cd63e1540cbb6bb4250f5f418be 2023-04-03 13:59:25,209 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/table/48ec9cd63e1540cbb6bb4250f5f418be, entries=9, sequenceid=93, filesize=5.5 K 2023-04-03 13:59:25,210 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 1268ms, sequenceid=93, compaction requested=false 2023-04-03 13:59:25,216 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-03 13:59:25,217 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-03 13:59:25,218 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-03 13:59:25,218 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-03 13:59:25,219 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-03 13:59:25,346 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,39695,1680530174856; all regions closed. 2023-04-03 13:59:25,360 DEBUG [RS:0;jenkins-hbase20:39695] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/oldWALs 2023-04-03 13:59:25,360 INFO [RS:0;jenkins-hbase20:39695] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C39695%2C1680530174856.meta:.meta(num 1680530177029) 2023-04-03 13:59:25,364 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/WALs/jenkins-hbase20.apache.org,39695,1680530174856/jenkins-hbase20.apache.org%2C39695%2C1680530174856.1680530176449 not finished, retry = 0 2023-04-03 13:59:25,468 DEBUG [RS:0;jenkins-hbase20:39695] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/oldWALs 2023-04-03 13:59:25,468 INFO [RS:0;jenkins-hbase20:39695] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C39695%2C1680530174856:(num 1680530176449) 2023-04-03 13:59:25,468 DEBUG [RS:0;jenkins-hbase20:39695] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:25,468 INFO [RS:0;jenkins-hbase20:39695] regionserver.LeaseManager(133): Closed leases 2023-04-03 13:59:25,469 INFO [RS:0;jenkins-hbase20:39695] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-03 13:59:25,469 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-03 13:59:25,471 INFO [RS:0;jenkins-hbase20:39695] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:39695 2023-04-03 13:59:25,479 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 13:59:25,479 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,39695,1680530174856 2023-04-03 13:59:25,479 ERROR [Listener at localhost.localdomain/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@687a0a3d rejected from java.util.concurrent.ThreadPoolExecutor@6c4583ea[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2264] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 13:59:25,480 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 13:59:25,480 ERROR [Listener at localhost.localdomain/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@20b7c48c rejected from java.util.concurrent.ThreadPoolExecutor@6c4583ea[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2264] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 13:59:25,481 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,39695,1680530174856] 2023-04-03 13:59:25,482 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,39695,1680530174856; numProcessing=1 2023-04-03 13:59:25,483 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,39695,1680530174856 already deleted, retry=false 2023-04-03 13:59:25,483 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,39695,1680530174856 expired; onlineServers=0 2023-04-03 13:59:25,483 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,37337,1680530173952' ***** 2023-04-03 13:59:25,483 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-03 13:59:25,483 DEBUG [M:0;jenkins-hbase20:37337] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@729188b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-04-03 13:59:25,483 INFO [M:0;jenkins-hbase20:37337] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,37337,1680530173952 2023-04-03 13:59:25,483 INFO [M:0;jenkins-hbase20:37337] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,37337,1680530173952; all regions closed. 2023-04-03 13:59:25,483 DEBUG [M:0;jenkins-hbase20:37337] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 13:59:25,483 DEBUG [M:0;jenkins-hbase20:37337] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-03 13:59:25,483 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-03 13:59:25,483 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680530176167] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1680530176167,5,FailOnTimeoutGroup] 2023-04-03 13:59:25,483 DEBUG [M:0;jenkins-hbase20:37337] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-03 13:59:25,483 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680530176167] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1680530176167,5,FailOnTimeoutGroup] 2023-04-03 13:59:25,484 INFO [M:0;jenkins-hbase20:37337] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-03 13:59:25,484 INFO [M:0;jenkins-hbase20:37337] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-03 13:59:25,484 INFO [M:0;jenkins-hbase20:37337] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-04-03 13:59:25,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-03 13:59:25,484 DEBUG [M:0;jenkins-hbase20:37337] master.HMaster(1512): Stopping service threads 2023-04-03 13:59:25,484 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 13:59:25,484 INFO [M:0;jenkins-hbase20:37337] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-03 13:59:25,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 13:59:25,485 ERROR [M:0;jenkins-hbase20:37337] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (346667566) connection to localhost.localdomain/127.0.0.1:35003 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-03 13:59:25,485 INFO [M:0;jenkins-hbase20:37337] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-03 13:59:25,485 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-03 13:59:25,485 DEBUG [M:0;jenkins-hbase20:37337] zookeeper.ZKUtil(398): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-03 13:59:25,485 WARN [M:0;jenkins-hbase20:37337] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-03 13:59:25,485 INFO [M:0;jenkins-hbase20:37337] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-03 13:59:25,486 INFO [M:0;jenkins-hbase20:37337] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-03 13:59:25,486 DEBUG [M:0;jenkins-hbase20:37337] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-03 13:59:25,486 INFO [M:0;jenkins-hbase20:37337] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 13:59:25,486 DEBUG [M:0;jenkins-hbase20:37337] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 13:59:25,486 DEBUG [M:0;jenkins-hbase20:37337] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-03 13:59:25,486 DEBUG [M:0;jenkins-hbase20:37337] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 13:59:25,486 INFO [M:0;jenkins-hbase20:37337] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.35 KB heapSize=393.97 KB 2023-04-03 13:59:25,495 INFO [M:0;jenkins-hbase20:37337] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.35 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/063145c9e394404dac1eaed81b4a09ba 2023-04-03 13:59:25,498 INFO [M:0;jenkins-hbase20:37337] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 063145c9e394404dac1eaed81b4a09ba 2023-04-03 13:59:25,499 DEBUG [M:0;jenkins-hbase20:37337] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/063145c9e394404dac1eaed81b4a09ba as hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/063145c9e394404dac1eaed81b4a09ba 2023-04-03 13:59:25,502 INFO [M:0;jenkins-hbase20:37337] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 063145c9e394404dac1eaed81b4a09ba 2023-04-03 13:59:25,502 INFO [M:0;jenkins-hbase20:37337] regionserver.HStore(1080): Added hdfs://localhost.localdomain:35003/user/jenkins/test-data/9ca51d18-deb3-d320-9499-c548a2988156/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/063145c9e394404dac1eaed81b4a09ba, entries=65, sequenceid=578, filesize=24.7 K 2023-04-03 13:59:25,503 INFO [M:0;jenkins-hbase20:37337] regionserver.HRegion(2948): Finished flush of dataSize ~345.35 KB/353642, heapSize ~393.95 KB/403408, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 17ms, sequenceid=578, compaction requested=false 2023-04-03 13:59:25,503 INFO [M:0;jenkins-hbase20:37337] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 13:59:25,504 DEBUG [M:0;jenkins-hbase20:37337] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 13:59:25,506 INFO [M:0;jenkins-hbase20:37337] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-03 13:59:25,506 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-03 13:59:25,507 INFO [M:0;jenkins-hbase20:37337] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:37337 2023-04-03 13:59:25,507 DEBUG [M:0;jenkins-hbase20:37337] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,37337,1680530173952 already deleted, retry=false 2023-04-03 13:59:25,582 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 13:59:25,582 INFO [RS:0;jenkins-hbase20:39695] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,39695,1680530174856; zookeeper connection closed. 2023-04-03 13:59:25,583 ERROR [Listener at localhost.localdomain/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@583e9378 rejected from java.util.concurrent.ThreadPoolExecutor@6c4583ea[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2264] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 13:59:25,585 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): regionserver:39695-0x100771d85340001, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 13:59:25,585 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3c0bf1b9] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3c0bf1b9 2023-04-03 13:59:25,585 ERROR [Listener at localhost.localdomain/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5765d14c rejected from java.util.concurrent.ThreadPoolExecutor@6c4583ea[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2264] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 13:59:25,586 INFO [Listener at localhost.localdomain/43477] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-03 13:59:25,683 INFO [M:0;jenkins-hbase20:37337] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,37337,1680530173952; zookeeper connection closed. 2023-04-03 13:59:25,683 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 13:59:25,683 ERROR [Listener at localhost.localdomain/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2bed84ad rejected from java.util.concurrent.ThreadPoolExecutor@ab91574[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3790] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 13:59:25,683 DEBUG [Listener at localhost.localdomain/43477-EventThread] zookeeper.ZKWatcher(600): master:37337-0x100771d85340000, quorum=127.0.0.1:57081, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 13:59:25,683 ERROR [Listener at localhost.localdomain/43477-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@14e2d116 rejected from java.util.concurrent.ThreadPoolExecutor@ab91574[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3790] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 13:59:25,686 WARN [Listener at localhost.localdomain/43477] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-03 13:59:25,691 INFO [Listener at localhost.localdomain/43477] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-03 13:59:25,799 WARN [BP-1153156962-148.251.75.209-1680530171395 heartbeating to localhost.localdomain/127.0.0.1:35003] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-03 13:59:25,799 WARN [BP-1153156962-148.251.75.209-1680530171395 heartbeating to localhost.localdomain/127.0.0.1:35003] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1153156962-148.251.75.209-1680530171395 (Datanode Uuid 5db87ce9-8a9b-49d5-8f71-6cbe87f3a0d0) service to localhost.localdomain/127.0.0.1:35003 2023-04-03 13:59:25,803 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/cluster_f53e868d-306e-b331-2b8a-fd0a76147c40/dfs/data/data1/current/BP-1153156962-148.251.75.209-1680530171395] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-03 13:59:25,804 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/d59de7ae-bf20-c35b-a0cf-3cf109cedd27/cluster_f53e868d-306e-b331-2b8a-fd0a76147c40/dfs/data/data2/current/BP-1153156962-148.251.75.209-1680530171395] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-03 13:59:25,837 INFO [Listener at localhost.localdomain/43477] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-04-03 13:59:25,988 INFO [Listener at localhost.localdomain/43477] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-03 13:59:26,131 INFO [Listener at localhost.localdomain/43477] hbase.HBaseTestingUtility(1293): Minicluster is down